[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:36:35 dist-test-slave-rgb1 krb5kdc[29652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:36:35 dist-test-slave-rgb1 krb5kdc[29652](info): set up 2 sockets
Aug 09 22:36:35 dist-test-slave-rgb1 krb5kdc[29652](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 22:36:36.291841 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.010s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:36:36 dist-test-slave-rgb1 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:36:36.530545 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40257
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:40257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:36.926301 29665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:36.926720 29665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:36.926891 29665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:36.935127 29665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:36:36.935271 29665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:36.935367 29665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:36.935496 29665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:36:36.935732 29665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:36:36.948519 29665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:40257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:36.950721 29665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:36.952868 29665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:36.980603 29670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:36.980310 29671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:36.980449 29673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:36.982426 29665 server_base.cc:1034] running on GCE node
I20260809 22:36:36.983731 29665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:36.986222 29665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:36.987713 29665 hybrid_clock.cc:648] HybridClock initialized: now 1786314996987587 us; error 92 us; skew 500 ppm
I20260809 22:36:36.988471 29665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:36 dist-test-slave-rgb1 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314996, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:36.993175 29665 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:36:36.996126 29665 webserver.cc:460] Webserver started at http://127.28.243.126:39113/ using document root <none> and password file <none>
I20260809 22:36:36.997447 29665 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:36.997627 29665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:36.998152 29665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:37.002912 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "14d3b3aca24242e4b63c814f49c31c40"
format_stamp: "Formatted at 2026-08-09 22:36:36 on dist-test-slave-rgb1"
server_key: "41e06eb5aedee7cfde06fa68b4d4bc4a"
server_key_iv: "06a41a05a39d3982aa5d6fde39772269"
server_key_version: "encryptionkey@0"
I20260809 22:36:37.004473 29665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "14d3b3aca24242e4b63c814f49c31c40"
format_stamp: "Formatted at 2026-08-09 22:36:36 on dist-test-slave-rgb1"
server_key: "41e06eb5aedee7cfde06fa68b4d4bc4a"
server_key_iv: "06a41a05a39d3982aa5d6fde39772269"
server_key_version: "encryptionkey@0"
I20260809 22:36:37.014384 29665 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.005s
I20260809 22:36:37.021083 29680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:37.023519 29665 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260809 22:36:37.023896 29665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "14d3b3aca24242e4b63c814f49c31c40"
format_stamp: "Formatted at 2026-08-09 22:36:36 on dist-test-slave-rgb1"
server_key: "41e06eb5aedee7cfde06fa68b4d4bc4a"
server_key_iv: "06a41a05a39d3982aa5d6fde39772269"
server_key_version: "encryptionkey@0"
I20260809 22:36:37.024163 29665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:37.045169 29665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:37.051988 29665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:37.052510 29665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:37.084205 29665 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:40257
I20260809 22:36:37.084257 29737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:40257 every 8 connection(s)
I20260809 22:36:37.086939 29665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:36:37.094417 29738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:37.097263 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 29665
I20260809 22:36:37.097704 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:36:37.100167 29645 external_mini_cluster.cc:1428] Setting key 6bca449f84f4cde5f42cd0429efe9660
I20260809 22:36:37.117769 29738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40: Bootstrap starting.
I20260809 22:36:37.127111 29738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:37.129681 29738 log.cc:824] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:37.135710 29738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40: No bootstrap required, opened a new log
Aug 09 22:36:37 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:37.149669 29738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d3b3aca24242e4b63c814f49c31c40" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40257 } }
I20260809 22:36:37.150312 29738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:37.150569 29738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d3b3aca24242e4b63c814f49c31c40, State: Initialized, Role: FOLLOWER
I20260809 22:36:37.151793 29738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d3b3aca24242e4b63c814f49c31c40" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40257 } }
I20260809 22:36:37.152189 29738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:36:37.152541 29738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:36:37.153031 29738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:37.156072 29738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d3b3aca24242e4b63c814f49c31c40" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40257 } }
I20260809 22:36:37.157362 29738 leader_election.cc:302] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d3b3aca24242e4b63c814f49c31c40; no voters:
I20260809 22:36:37.158567 29738 leader_election.cc:288] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:36:37.158921 29744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:37.160462 29744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [term 1 LEADER]: Becoming Leader. State: Replica: 14d3b3aca24242e4b63c814f49c31c40, State: Running, Role: LEADER
I20260809 22:36:37.161502 29744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d3b3aca24242e4b63c814f49c31c40" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40257 } }
I20260809 22:36:37.162659 29738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:36:37.167677 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:37.109881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.28.243.126:40257)
0809 22:36:37.113045 (+ 3164us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:37.113060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:37.113114 (+ 54us) server_negotiation.cc:407] Connection header received
0809 22:36:37.128640 (+ 15526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:37.128682 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:37.129381 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:37.129879 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:37.135963 (+ 6084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:37.138308 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:37.141648 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:37.142616 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:37.142859 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:37.151883 (+ 9024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:37.151958 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:37.152006 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:37.152071 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:37.156093 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:37.157699 (+ 1606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:37.157708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:37.157714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:37.158060 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:37.158604 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:37.158611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:37.158618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:37.159803 (+ 1185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:37.161418 (+ 1615us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:37.161790 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 22:36:37.163323 (+ 1533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2706,"thread_start_us":2149,"threads_started":1}
I20260809 22:36:37.172891 29745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d3b3aca24242e4b63c814f49c31c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d3b3aca24242e4b63c814f49c31c40" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40257 } } }
I20260809 22:36:37.173774 29745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:37.174592 29746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d3b3aca24242e4b63c814f49c31c40. Latest consensus state: current_term: 1 leader_uuid: "14d3b3aca24242e4b63c814f49c31c40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d3b3aca24242e4b63c814f49c31c40" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40257 } } }
I20260809 22:36:37.175348 29746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40 [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:37.179435 29751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:36:37.189823 29751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:36:37.206745 29751 catalog_manager.cc:1347] Generated new cluster ID: fee1d4c84ad644ee839c4d927d38f418
I20260809 22:36:37.206948 29751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:36:37.218633 29751 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:36:37.220799 29751 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:36:37.229691 29751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d3b3aca24242e4b63c814f49c31c40: Generated new TSK 0
I20260809 22:36:37.230901 29751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:36:37.293919 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40257
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:37.683683 29765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:37.684072 29765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:37.684247 29765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:37.693200 29765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:37.693341 29765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:37.693576 29765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:36:37.705025 29765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:37.706504 29765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:37.708055 29765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:37.722997 29770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:37.722934 29773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:37.722920 29771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:37.724315 29765 server_base.cc:1034] running on GCE node
I20260809 22:36:37.725179 29765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:37.726337 29765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:37.727591 29765 hybrid_clock.cc:648] HybridClock initialized: now 1786314997727547 us; error 40 us; skew 500 ppm
I20260809 22:36:37.728081 29765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:37 dist-test-slave-rgb1 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:37.732218 29765 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:36:37.734419 29765 webserver.cc:460] Webserver started at http://127.28.243.65:44025/ using document root <none> and password file <none>
I20260809 22:36:37.735195 29765 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:37.735324 29765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:37.735623 29765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:37.739245 29765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "f8b83dbab66a41cb8489f7527c4231cb"
format_stamp: "Formatted at 2026-08-09 22:36:37 on dist-test-slave-rgb1"
server_key: "2efbade4241699cda3a887ac27b2c15b"
server_key_iv: "cb24260d4a6a4570e9d9b60f1218245b"
server_key_version: "encryptionkey@0"
I20260809 22:36:37.740223 29765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "f8b83dbab66a41cb8489f7527c4231cb"
format_stamp: "Formatted at 2026-08-09 22:36:37 on dist-test-slave-rgb1"
server_key: "2efbade4241699cda3a887ac27b2c15b"
server_key_iv: "cb24260d4a6a4570e9d9b60f1218245b"
server_key_version: "encryptionkey@0"
I20260809 22:36:37.746984 29765 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:36:37.751304 29780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:37.753103 29765 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:36:37.753269 29765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "f8b83dbab66a41cb8489f7527c4231cb"
format_stamp: "Formatted at 2026-08-09 22:36:37 on dist-test-slave-rgb1"
server_key: "2efbade4241699cda3a887ac27b2c15b"
server_key_iv: "cb24260d4a6a4570e9d9b60f1218245b"
server_key_version: "encryptionkey@0"
I20260809 22:36:37.753481 29765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:37.791652 29765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:37.797210 29765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:37.797520 29765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:37.798821 29765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:36:37.800832 29765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:37.800922 29765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:37.801132 29765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:37.801199 29765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:37.846261 29765 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:38449
I20260809 22:36:37.846269 29902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:38449 every 8 connection(s)
I20260809 22:36:37.847743 29765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:36:37.858094 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 29765
I20260809 22:36:37.858395 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:36:37.859071 29645 external_mini_cluster.cc:1428] Setting key 04d187ce0e3cb3e78982ad860d98eb71
Aug 09 22:36:37 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314997, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:37.876197 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:37.852751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:46249 (local address 127.28.243.126:40257)
0809 22:36:37.853367 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:37.853372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:37.854232 (+ 860us) server_negotiation.cc:407] Connection header received
0809 22:36:37.856274 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:37.856280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:37.856384 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:37.856513 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:37.858683 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:37.859717 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:37.862541 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:37.863062 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:37.863184 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:37.869296 (+ 6112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:37.869302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:37.869305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:37.869345 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:37.872508 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:37.873461 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:37.873474 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:37.873477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:37.873622 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:37.874124 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:37.874129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:37.874131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:37.874478 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:37.874651 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:37.875539 (+ 888us) server_negotiation.cc:299] Negotiation successful
0809 22:36:37.875804 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":367,"threads_started":1}
I20260809 22:36:37.877285 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:37.853251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40257 (local address 127.28.243.65:46249)
0809 22:36:37.854044 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:37.854085 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:37.855977 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:37.856796 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:37.856805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:37.857459 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:37.858403 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:37.858445 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:37.859974 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:37.859979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:37.862299 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:37.862321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:37.863451 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:37.863464 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:37.863952 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:37.865173 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:37.865245 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:37.868987 (+ 3742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:37.872753 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:37.872764 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:37.872786 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:37.873308 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:37.873750 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:37.873755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:37.873759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:37.873981 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:37.874621 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:37.874630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:37.875020 (+ 390us) client_negotiation.cc:769] Sending connection context
0809 22:36:37.875369 (+ 349us) client_negotiation.cc:240] Negotiation successful
0809 22:36:37.875982 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":326,"threads_started":1}
I20260809 22:36:37.879087 29903 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40257
I20260809 22:36:37.879614 29903 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:37.880520 29903 heartbeater.cc:511] Master 127.28.243.126:40257 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:36:37.884778 29698 ts_manager.cc:194] Registered new tserver with Master: f8b83dbab66a41cb8489f7527c4231cb (127.28.243.65:38449)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
I20260809 22:36:37.887236 29698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:46249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:36:37.913241 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40257
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:38.287875 29909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:38.288218 29909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:38.288439 29909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:38.296715 29909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:38.296844 29909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:38.297065 29909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:36:38.307690 29909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:38.309552 29909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:38.311506 29909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:38.327333 29914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:38.327272 29915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:38.327504 29917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:38.334749 29909 server_base.cc:1034] running on GCE node
I20260809 22:36:38.335538 29909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:38.336802 29909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:38.338084 29909 hybrid_clock.cc:648] HybridClock initialized: now 1786314998338001 us; error 41 us; skew 500 ppm
I20260809 22:36:38.338435 29909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:38 dist-test-slave-rgb1 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:38.342119 29909 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:36:38.343979 29909 webserver.cc:460] Webserver started at http://127.28.243.66:45879/ using document root <none> and password file <none>
I20260809 22:36:38.344849 29909 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:38.345006 29909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:38.345350 29909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:38.348676 29909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "ac9ac4b4d3d04b359c2f20aee966059f"
format_stamp: "Formatted at 2026-08-09 22:36:38 on dist-test-slave-rgb1"
server_key: "2847218900856b38818f66bb860777ff"
server_key_iv: "a322e94dbdc0031f767f78e78f00dcc2"
server_key_version: "encryptionkey@0"
I20260809 22:36:38.349548 29909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "ac9ac4b4d3d04b359c2f20aee966059f"
format_stamp: "Formatted at 2026-08-09 22:36:38 on dist-test-slave-rgb1"
server_key: "2847218900856b38818f66bb860777ff"
server_key_iv: "a322e94dbdc0031f767f78e78f00dcc2"
server_key_version: "encryptionkey@0"
I20260809 22:36:38.355445 29909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 22:36:38.359444 29924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:38.361080 29909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:36:38.361289 29909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "ac9ac4b4d3d04b359c2f20aee966059f"
format_stamp: "Formatted at 2026-08-09 22:36:38 on dist-test-slave-rgb1"
server_key: "2847218900856b38818f66bb860777ff"
server_key_iv: "a322e94dbdc0031f767f78e78f00dcc2"
server_key_version: "encryptionkey@0"
I20260809 22:36:38.361512 29909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:38.381398 29909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:38.386466 29909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:38.386797 29909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:38.387925 29909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:36:38.389585 29909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:38.389699 29909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:38.389878 29909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:38.389956 29909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:38.434096 29909 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39385
I20260809 22:36:38.434139 30045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39385 every 8 connection(s)
I20260809 22:36:38.435680 29909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:36:38.436075 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 29909
I20260809 22:36:38.436306 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:36:38.437007 29645 external_mini_cluster.cc:1428] Setting key 026d0ba32aaf4112aba54c91ac2d5dd5
Aug 09 22:36:38 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
I20260809 22:36:38.463574 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:38.438527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:47031 (local address 127.28.243.126:40257)
0809 22:36:38.439140 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:38.439148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:38.440409 (+ 1261us) server_negotiation.cc:407] Connection header received
0809 22:36:38.442558 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:38.442567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:38.442668 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:38.442799 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:38.444755 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:38.445882 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:38.448833 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:38.449377 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:38.449512 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:38.455001 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:38.455008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:38.455011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:38.455152 (+ 141us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:38.459209 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:38.460307 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:38.460314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:38.460317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:38.460509 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:38.461339 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:38.461344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:38.461346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:38.461697 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:38.461839 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:38.462859 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0809 22:36:38.463195 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":295,"threads_started":1}
I20260809 22:36:38.465149 30049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:38.439263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40257 (local address 127.28.243.66:47031)
0809 22:36:38.440191 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:38.440250 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:38.442203 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:38.443122 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:38.443133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:38.443756 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:38.444390 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:38.444408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:38.446091 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:38.446099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:38.448630 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:38.448653 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:38.449654 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:38.449664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:38.450041 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:38.450946 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:38.450989 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:38.454651 (+ 3662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:38.459427 (+ 4776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:38.459463 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:38.459487 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:38.460036 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:38.460814 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:38.460818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:38.460821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:38.461106 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:38.461860 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:38.461868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:38.462292 (+ 424us) client_negotiation.cc:769] Sending connection context
0809 22:36:38.462611 (+ 319us) client_negotiation.cc:240] Negotiation successful
0809 22:36:38.463237 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":414,"threads_started":1}
I20260809 22:36:38.467604 30046 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40257
I20260809 22:36:38.468439 30046 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:38.469410 30046 heartbeater.cc:511] Master 127.28.243.126:40257 requested a full tablet report, sending...
I20260809 22:36:38.471637 29698 ts_manager.cc:194] Registered new tserver with Master: ac9ac4b4d3d04b359c2f20aee966059f (127.28.243.66:39385)
I20260809 22:36:38.472677 29698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:47031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:36:38.494875 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40257
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:38.866500 30052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:38.866892 30052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:38.867049 30052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:38.874739 30052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:38.874931 30052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:38.875138 30052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:36:38.885591 30052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:38.886976 30052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:38.888362 30052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260809 22:36:38.891359 29903 heartbeater.cc:503] Master 127.28.243.126:40257 was elected leader, sending a full tablet report...
W20260809 22:36:38.902796 30060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:38.902876 30057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:38.902872 30058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:38.903975 30052 server_base.cc:1034] running on GCE node
I20260809 22:36:38.904760 30052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:38.905793 30052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:38.907025 30052 hybrid_clock.cc:648] HybridClock initialized: now 1786314998906958 us; error 49 us; skew 500 ppm
I20260809 22:36:38.907327 30052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:38 dist-test-slave-rgb1 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:38.911302 30052 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:36:38.913311 30052 webserver.cc:460] Webserver started at http://127.28.243.67:33225/ using document root <none> and password file <none>
I20260809 22:36:38.914090 30052 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:38.914242 30052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:38.914649 30052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:38.917958 30052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "2a86fd17e4644dd3afae2532abcefc01"
format_stamp: "Formatted at 2026-08-09 22:36:38 on dist-test-slave-rgb1"
server_key: "eb3a4c60343841bb40bd4d179976a608"
server_key_iv: "918ef0e33c78de23c8db2929fb1e63a0"
server_key_version: "encryptionkey@0"
I20260809 22:36:38.918788 30052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "2a86fd17e4644dd3afae2532abcefc01"
format_stamp: "Formatted at 2026-08-09 22:36:38 on dist-test-slave-rgb1"
server_key: "eb3a4c60343841bb40bd4d179976a608"
server_key_iv: "918ef0e33c78de23c8db2929fb1e63a0"
server_key_version: "encryptionkey@0"
I20260809 22:36:38.924813 30052 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 22:36:38.928725 30067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:38.930393 30052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:36:38.930647 30052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "2a86fd17e4644dd3afae2532abcefc01"
format_stamp: "Formatted at 2026-08-09 22:36:38 on dist-test-slave-rgb1"
server_key: "eb3a4c60343841bb40bd4d179976a608"
server_key_iv: "918ef0e33c78de23c8db2929fb1e63a0"
server_key_version: "encryptionkey@0"
I20260809 22:36:38.930887 30052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:38.955436 30052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:38.961287 30052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:38.961638 30052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:38.962731 30052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:36:38.964349 30052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:38.964464 30052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:38.964653 30052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:38.964731 30052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:39.010497 30052 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:33877
I20260809 22:36:39.010600 30189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:33877 every 8 connection(s)
I20260809 22:36:39.011963 30052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:36:39.018285 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 30052
I20260809 22:36:39.018522 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:36:39.019124 29645 external_mini_cluster.cc:1428] Setting key c110664a1e126b916a97673db35c8c22
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314998, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:39.039382 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:39.015797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:47481 (local address 127.28.243.126:40257)
0809 22:36:39.016491 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:39.016500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:39.017825 (+ 1325us) server_negotiation.cc:407] Connection header received
0809 22:36:39.020651 (+ 2826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:39.020658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:39.020749 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:39.020861 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:39.023160 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.024354 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.026235 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.026736 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.026906 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:39.031478 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:39.031484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:39.031488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:39.031527 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:39.034808 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.036193 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.036202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.036206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.036405 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.037303 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.037307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.037309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.037668 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:39.037874 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:39.038701 (+ 827us) server_negotiation.cc:299] Negotiation successful
0809 22:36:39.038958 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":367,"threads_started":1}
I20260809 22:36:39.040550 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:39.016431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40257 (local address 127.28.243.67:47481)
0809 22:36:39.017606 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:39.017650 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:39.020333 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:39.021172 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:39.021182 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:39.022087 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:39.022911 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:39.022940 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.024542 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.024548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:39.026055 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:39.026070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.027024 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.027037 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:39.027372 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:39.028230 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:39.028258 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:39.031205 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:39.035008 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:39.035021 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:39.035042 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:39.035854 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:39.036752 (+ 898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:39.036757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:39.036761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:39.037142 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:39.037850 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:39.037861 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:39.038279 (+ 418us) client_negotiation.cc:769] Sending connection context
0809 22:36:39.038600 (+ 321us) client_negotiation.cc:240] Negotiation successful
0809 22:36:39.039279 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":376,"threads_started":1}
I20260809 22:36:39.042217 30190 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40257
I20260809 22:36:39.042656 30190 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:39.043453 30190 heartbeater.cc:511] Master 127.28.243.126:40257 requested a full tablet report, sending...
I20260809 22:36:39.045254 29698 ts_manager.cc:194] Registered new tserver with Master: 2a86fd17e4644dd3afae2532abcefc01 (127.28.243.67:33877)
I20260809 22:36:39.046197 29698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:47481
I20260809 22:36:39.051050 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:36:39.113024 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:39.092669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.28.243.65:38449)
0809 22:36:39.093979 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:39.093987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:39.094014 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:36:39.094078 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:39.094084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:39.094322 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:39.094528 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:39.096394 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.097683 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.099212 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.100048 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.100225 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:39.105077 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:39.105093 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:39.105109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:39.105153 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:39.109504 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.110345 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.110353 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.110356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.110501 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.111005 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.111010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.111013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.111512 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:39.111885 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:39.112248 (+ 363us) server_negotiation.cc:299] Negotiation successful
0809 22:36:39.112594 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":936,"threads_started":1}
W20260809 22:36:39.114528 29811 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51376
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[29652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:36:39.175879 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:39.156674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.28.243.65:38449)
0809 22:36:39.156923 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:39.156935 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:39.157192 (+ 257us) server_negotiation.cc:407] Connection header received
0809 22:36:39.157383 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:39.157388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:39.157497 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:39.157625 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:39.159722 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.160431 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.162222 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.162720 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.162918 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:39.168746 (+ 5828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:39.168751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:39.168755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:39.168799 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:39.172813 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.173700 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.173704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.173707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.173953 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.174415 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.174419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.174421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.174713 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:39.175008 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:39.175290 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 22:36:39.175489 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 22:36:39.178563 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 29765
I20260809 22:36:39.193553 30199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:39.180041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.28.243.65:38449)
0809 22:36:39.180319 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:39.180326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:39.180672 (+ 346us) server_negotiation.cc:407] Connection header received
0809 22:36:39.180887 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:39.180891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:39.180964 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:39.181072 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:39.182422 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.183226 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.184698 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.185270 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.185378 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:39.187848 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:39.187853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:39.187856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:39.187886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:39.190661 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.191365 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.191370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.191373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.191566 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.192029 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.192033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.192035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.192337 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:39.192691 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:39.192931 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 22:36:39.193101 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 22:36:39.194516 29898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:39.285454 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 29765
I20260809 22:36:39.304793 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 29909
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:36:39.328135 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:39.305961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.28.243.66:39385)
0809 22:36:39.306600 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:39.306607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:39.306633 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:36:39.306702 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:39.306709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:39.306943 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:39.307088 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:39.308360 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.309523 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.310835 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.311492 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.311605 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:39.320893 (+ 9288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:39.320913 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:39.320938 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:39.321032 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:39.324374 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.325404 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.325413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.325416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.325598 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.326160 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.326168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.326171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.326706 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:39.326993 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:39.327378 (+ 385us) server_negotiation.cc:299] Negotiation successful
0809 22:36:39.327643 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":367,"threads_started":1}
I20260809 22:36:39.329818 30041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:39.425884 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 29909
I20260809 22:36:39.444967 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30052
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[29652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786314999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:36:39.463975 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:39.446164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.28.243.67:33877)
0809 22:36:39.446834 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:39.446901 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:39.446936 (+ 35us) server_negotiation.cc:407] Connection header received
0809 22:36:39.447000 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:39.447006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:39.447240 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:39.447384 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:39.448945 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.450133 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.451564 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:39.452201 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:39.452312 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:39.457172 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:39.457187 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:39.457203 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:39.457249 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:39.460557 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.461388 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.461396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.461399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.461552 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:39.462049 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:39.462053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:39.462055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:39.462506 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:39.462849 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:39.463243 (+ 394us) server_negotiation.cc:299] Negotiation successful
0809 22:36:39.463554 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":351,"threads_started":1}
I20260809 22:36:39.465512 30185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:39.571259 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 30052
I20260809 22:36:39.598999 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 29665
I20260809 22:36:39.601249 29732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:39.745266 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 29665
2026-08-09T22:36:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4715 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[30214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[30214](info): set up 2 sockets
Aug 09 22:36:39 dist-test-slave-rgb1 krb5kdc[30214](info): commencing operation
krb5kdc: starting...
W20260809 22:36:40.979348 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260809 22:36:41.143249 29645 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.164s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260809 22:36:41.331792 29645 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.188s user 0.001s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260809 22:36:41.534210 29645 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.202s user 0.000s sys 0.006s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/test-user.keytab.
W20260809 22:36:41.946172 29645 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.412s user 0.000s sys 0.006s
Aug 09 22:36:41 dist-test-slave-rgb1 krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:36:42.058068 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39609
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:42.405988 30227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:42.406319 30227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:42.406412 30227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:42.414072 30227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:36:42.414201 30227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:42.414250 30227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:42.414301 30227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:36:42.414347 30227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:36:42.424641 30227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:42.426090 30227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:42.427393 30227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:42.440774 30232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:42.440806 30233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:42.441013 30235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:42.441926 30227 server_base.cc:1034] running on GCE node
I20260809 22:36:42.442832 30227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:42.444460 30227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:42.445770 30227 hybrid_clock.cc:648] HybridClock initialized: now 1786315002445717 us; error 41 us; skew 500 ppm
I20260809 22:36:42.446211 30227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:42 dist-test-slave-rgb1 krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315002, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:42.450404 30227 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:36:42.452551 30227 webserver.cc:460] Webserver started at http://127.28.243.126:38749/ using document root <none> and password file <none>
I20260809 22:36:42.453341 30227 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:42.453456 30227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:42.453754 30227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:42.457351 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "0c7ead30cbf24f66ae2d721b12436f10"
format_stamp: "Formatted at 2026-08-09 22:36:42 on dist-test-slave-rgb1"
server_key: "5b16601338c29d6736ca23602b1a979c"
server_key_iv: "0cf02ce622b0d7851b6d851d2fadbcb8"
server_key_version: "encryptionkey@0"
I20260809 22:36:42.458381 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "0c7ead30cbf24f66ae2d721b12436f10"
format_stamp: "Formatted at 2026-08-09 22:36:42 on dist-test-slave-rgb1"
server_key: "5b16601338c29d6736ca23602b1a979c"
server_key_iv: "0cf02ce622b0d7851b6d851d2fadbcb8"
server_key_version: "encryptionkey@0"
I20260809 22:36:42.464659 30227 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 22:36:42.468803 30243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:42.470439 30227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:36:42.470659 30227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "0c7ead30cbf24f66ae2d721b12436f10"
format_stamp: "Formatted at 2026-08-09 22:36:42 on dist-test-slave-rgb1"
server_key: "5b16601338c29d6736ca23602b1a979c"
server_key_iv: "0cf02ce622b0d7851b6d851d2fadbcb8"
server_key_version: "encryptionkey@0"
I20260809 22:36:42.470880 30227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:42.496987 30227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:42.502012 30227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:42.502337 30227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:42.522261 30227 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:39609
I20260809 22:36:42.522300 30298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:39609 every 8 connection(s)
I20260809 22:36:42.523758 30227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:36:42.527932 30300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:42.530028 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 30227
I20260809 22:36:42.530314 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:36:42.531104 29645 external_mini_cluster.cc:1428] Setting key 713c4a3912e8b74d1ce0094a0130bdb6
I20260809 22:36:42.540858 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10: Bootstrap starting.
I20260809 22:36:42.545079 30300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:42.546568 30300 log.cc:824] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10: Log is configured to *not* fsync() on all Append() calls
Aug 09 22:36:42 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:42.550414 30300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10: No bootstrap required, opened a new log
I20260809 22:36:42.560853 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:42.533381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.28.243.126:39609)
0809 22:36:42.534288 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:42.534301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:42.534353 (+ 52us) server_negotiation.cc:407] Connection header received
0809 22:36:42.536187 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:42.536212 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:42.536758 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:42.537167 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:42.539144 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:42.540715 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:42.542536 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:42.543636 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:42.543788 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:42.549559 (+ 5771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:42.549570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:42.549586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:42.549629 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:42.554686 (+ 5057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:42.555721 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:42.555736 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:42.555746 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:42.556022 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:42.556501 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:42.556508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:42.556512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:42.557435 (+ 923us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:42.557773 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:42.558612 (+ 839us) server_negotiation.cc:299] Negotiation successful
0809 22:36:42.559160 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":432,"threads_started":1}
I20260809 22:36:42.562166 30300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7ead30cbf24f66ae2d721b12436f10" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39609 } }
I20260809 22:36:42.562717 30300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:42.562832 30300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7ead30cbf24f66ae2d721b12436f10, State: Initialized, Role: FOLLOWER
I20260809 22:36:42.563539 30300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7ead30cbf24f66ae2d721b12436f10" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39609 } }
I20260809 22:36:42.563808 30300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:36:42.563949 30300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:36:42.564146 30300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:42.566567 30300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7ead30cbf24f66ae2d721b12436f10" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39609 } }
I20260809 22:36:42.567113 30300 leader_election.cc:302] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7ead30cbf24f66ae2d721b12436f10; no voters:
I20260809 22:36:42.567945 30300 leader_election.cc:288] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:36:42.568094 30306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:42.569039 30306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7ead30cbf24f66ae2d721b12436f10, State: Running, Role: LEADER
I20260809 22:36:42.569645 30306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7ead30cbf24f66ae2d721b12436f10" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39609 } }
I20260809 22:36:42.570712 30300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:36:42.574025 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7ead30cbf24f66ae2d721b12436f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7ead30cbf24f66ae2d721b12436f10" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39609 } } }
I20260809 22:36:42.574618 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:42.574054 30308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7ead30cbf24f66ae2d721b12436f10. Latest consensus state: current_term: 1 leader_uuid: "0c7ead30cbf24f66ae2d721b12436f10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7ead30cbf24f66ae2d721b12436f10" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39609 } } }
I20260809 22:36:42.574775 30308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10 [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:42.577236 30315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:36:42.582703 30315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:36:42.592905 30315 catalog_manager.cc:1347] Generated new cluster ID: 8640bdc13f804bb29190b36494a7dd61
I20260809 22:36:42.593047 30315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:36:42.608029 30315 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:36:42.609866 30315 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:36:42.626325 30315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c7ead30cbf24f66ae2d721b12436f10: Generated new TSK 0
I20260809 22:36:42.627313 30315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:36:42.686398 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39609
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:43.055267 30327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:43.055584 30327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:43.055686 30327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:43.063517 30327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:43.063644 30327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:43.063766 30327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:36:43.075196 30327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:43.076835 30327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:43.078276 30327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:43.093029 30334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:43.093184 30332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:43.093161 30336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:43.094079 30327 server_base.cc:1034] running on GCE node
I20260809 22:36:43.094725 30327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:43.095713 30327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:43.096920 30327 hybrid_clock.cc:648] HybridClock initialized: now 1786315003096817 us; error 91 us; skew 500 ppm
I20260809 22:36:43.097303 30327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:43 dist-test-slave-rgb1 krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:43.100795 30327 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:36:43.102523 30327 webserver.cc:460] Webserver started at http://127.28.243.65:40941/ using document root <none> and password file <none>
I20260809 22:36:43.103281 30327 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:43.103441 30327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:43.103790 30327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:43.107008 30327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "19f50e8183c34239ace6f753ccdb6da9"
format_stamp: "Formatted at 2026-08-09 22:36:43 on dist-test-slave-rgb1"
server_key: "afb0882ce7b26ee0383097abbb607691"
server_key_iv: "55a3e3fcfedc498dd9a8575d654252fc"
server_key_version: "encryptionkey@0"
I20260809 22:36:43.107852 30327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "19f50e8183c34239ace6f753ccdb6da9"
format_stamp: "Formatted at 2026-08-09 22:36:43 on dist-test-slave-rgb1"
server_key: "afb0882ce7b26ee0383097abbb607691"
server_key_iv: "55a3e3fcfedc498dd9a8575d654252fc"
server_key_version: "encryptionkey@0"
I20260809 22:36:43.113660 30327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260809 22:36:43.117838 30344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:43.119450 30327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:36:43.119673 30327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "19f50e8183c34239ace6f753ccdb6da9"
format_stamp: "Formatted at 2026-08-09 22:36:43 on dist-test-slave-rgb1"
server_key: "afb0882ce7b26ee0383097abbb607691"
server_key_iv: "55a3e3fcfedc498dd9a8575d654252fc"
server_key_version: "encryptionkey@0"
I20260809 22:36:43.119911 30327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:43.154675 30327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:43.160480 30327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:43.160835 30327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:43.161931 30327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:36:43.163435 30327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:43.163552 30327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:43.163705 30327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:43.163789 30327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:43.203285 30327 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:41391
I20260809 22:36:43.203338 30464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:41391 every 8 connection(s)
I20260809 22:36:43.204869 30327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:36:43.209553 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 30327
I20260809 22:36:43.209823 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:36:43.210506 29645 external_mini_cluster.cc:1428] Setting key 859aa206cd9844ca121abd81914a5cbb
Aug 09 22:36:43 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:43.228685 30467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:43.207742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:56541 (local address 127.28.243.126:39609)
0809 22:36:43.208256 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:43.208261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:43.209411 (+ 1150us) server_negotiation.cc:407] Connection header received
0809 22:36:43.211710 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:43.211717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:43.211823 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:43.211954 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:43.213753 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.214572 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.216341 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.216886 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.217046 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:43.221579 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:43.221584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:43.221588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:43.221623 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:43.224721 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:43.225801 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:43.225822 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:43.225828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:43.226003 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:43.226628 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:43.226632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:43.226635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:43.226920 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:43.227051 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:43.228110 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0809 22:36:43.228297 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260809 22:36:43.230189 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:43.208340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39609 (local address 127.28.243.65:56541)
0809 22:36:43.209218 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:43.209267 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:43.211411 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:43.212240 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:43.212252 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:43.212914 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:43.213556 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:43.213575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.214771 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.214776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:43.216174 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:43.216187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.217159 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.217170 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:43.217437 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:43.218218 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:43.218242 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:43.221242 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:43.224867 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:43.224879 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:43.224899 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:43.225603 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:43.226142 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:43.226147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:43.226151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:43.226445 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:43.227069 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:43.227081 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:43.227612 (+ 531us) client_negotiation.cc:769] Sending connection context
0809 22:36:43.227956 (+ 344us) client_negotiation.cc:240] Negotiation successful
0809 22:36:43.228630 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":378,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:36:43.232106 30465 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39609
I20260809 22:36:43.232379 30465 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:43.233282 30465 heartbeater.cc:511] Master 127.28.243.126:39609 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
I20260809 22:36:43.235785 30262 ts_manager.cc:194] Registered new tserver with Master: 19f50e8183c34239ace6f753ccdb6da9 (127.28.243.65:41391)
I20260809 22:36:43.238044 30262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:56541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:36:43.262537 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39609
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:43.618683 30471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:43.618985 30471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:43.619156 30471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:43.626808 30471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:43.626974 30471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:43.627174 30471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:36:43.638695 30471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:43.640210 30471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:43.641865 30471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:43.657658 30477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:43.657622 30478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:43.657697 30481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:43.659229 30471 server_base.cc:1034] running on GCE node
I20260809 22:36:43.660039 30471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:43.661351 30471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:43.662623 30471 hybrid_clock.cc:648] HybridClock initialized: now 1786315003662565 us; error 56 us; skew 500 ppm
I20260809 22:36:43.663139 30471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:43 dist-test-slave-rgb1 krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:43.666818 30471 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:36:43.668534 30471 webserver.cc:460] Webserver started at http://127.28.243.66:41187/ using document root <none> and password file <none>
I20260809 22:36:43.669374 30471 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:43.669544 30471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:43.669900 30471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:43.673398 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "68fc8387f6024a689b2aa28e0073d6ec"
format_stamp: "Formatted at 2026-08-09 22:36:43 on dist-test-slave-rgb1"
server_key: "e8c26177edebca076b575e991224030e"
server_key_iv: "22d8949f647eb3f78bb9ea9ffc98d9dd"
server_key_version: "encryptionkey@0"
I20260809 22:36:43.674386 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "68fc8387f6024a689b2aa28e0073d6ec"
format_stamp: "Formatted at 2026-08-09 22:36:43 on dist-test-slave-rgb1"
server_key: "e8c26177edebca076b575e991224030e"
server_key_iv: "22d8949f647eb3f78bb9ea9ffc98d9dd"
server_key_version: "encryptionkey@0"
I20260809 22:36:43.680612 30471 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 22:36:43.684856 30488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:43.686355 30471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:36:43.686560 30471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "68fc8387f6024a689b2aa28e0073d6ec"
format_stamp: "Formatted at 2026-08-09 22:36:43 on dist-test-slave-rgb1"
server_key: "e8c26177edebca076b575e991224030e"
server_key_iv: "22d8949f647eb3f78bb9ea9ffc98d9dd"
server_key_version: "encryptionkey@0"
I20260809 22:36:43.686800 30471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:43.726135 30471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:43.731194 30471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:43.731523 30471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:43.732635 30471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:36:43.734129 30471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:43.734233 30471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:43.734421 30471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:43.734475 30471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:43.771987 30471 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:36885
I20260809 22:36:43.772121 30606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:36885 every 8 connection(s)
I20260809 22:36:43.773394 30471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:36:43.775347 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 30471
I20260809 22:36:43.775588 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:36:43.776253 29645 external_mini_cluster.cc:1428] Setting key c2e84b5dc7c1e02d417d74b3380e2924
Aug 09 22:36:43 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315003, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
I20260809 22:36:43.800809 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:43.776842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:54789 (local address 127.28.243.126:39609)
0809 22:36:43.777381 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:43.777386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:43.778687 (+ 1301us) server_negotiation.cc:407] Connection header received
0809 22:36:43.782121 (+ 3434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:43.782127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:43.782271 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:43.782417 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:43.785028 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.785846 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.788094 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.788677 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.788839 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:43.793085 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:43.793095 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:43.793103 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:43.793155 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:43.797148 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:43.798062 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:43.798069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:43.798075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:43.798272 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:43.798754 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:43.798757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:43.798759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:43.799138 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:43.799307 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:43.800161 (+ 854us) server_negotiation.cc:299] Negotiation successful
0809 22:36:43.800368 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":286,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
I20260809 22:36:43.801882 30611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:43.777573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39609 (local address 127.28.243.66:54789)
0809 22:36:43.778464 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:43.778553 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:43.781677 (+ 3124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:43.782752 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:43.782770 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:43.783757 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:43.784765 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:43.784797 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.786149 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.786154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:43.787883 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:43.787898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:43.788959 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:43.788979 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:43.789248 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:43.790059 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:43.790092 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:43.792794 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:43.797321 (+ 4527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:43.797329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:43.797350 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:43.797856 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:43.798417 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:43.798421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:43.798423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:43.798624 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:43.799311 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:43.799323 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:43.799742 (+ 419us) client_negotiation.cc:769] Sending connection context
0809 22:36:43.799999 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 22:36:43.800611 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":363,"threads_started":1}
I20260809 22:36:43.803526 30608 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39609
I20260809 22:36:43.803894 30608 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:43.804940 30608 heartbeater.cc:511] Master 127.28.243.126:39609 requested a full tablet report, sending...
I20260809 22:36:43.806890 30262 ts_manager.cc:194] Registered new tserver with Master: 68fc8387f6024a689b2aa28e0073d6ec (127.28.243.66:36885)
I20260809 22:36:43.808151 30262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:54789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:36:43.828949 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39609
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:44.177723 30614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:44.178068 30614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:44.178233 30614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:44.186000 30614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:44.186126 30614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:44.186312 30614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:36:44.196897 30614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39609
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:44.198253 30614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:44.199708 30614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:44.215212 30619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:44.215180 30620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:44.215384 30622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:44.216152 30614 server_base.cc:1034] running on GCE node
I20260809 22:36:44.216933 30614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:44.217958 30614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:44.219252 30614 hybrid_clock.cc:648] HybridClock initialized: now 1786315004219182 us; error 55 us; skew 500 ppm
I20260809 22:36:44.219653 30614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:44 dist-test-slave-rgb1 krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:44.223347 30614 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:36:44.225318 30614 webserver.cc:460] Webserver started at http://127.28.243.67:46433/ using document root <none> and password file <none>
I20260809 22:36:44.226109 30614 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:44.226279 30614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:44.226649 30614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:44.230000 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "958be0fda3c241378910ca8965c5cbba"
format_stamp: "Formatted at 2026-08-09 22:36:44 on dist-test-slave-rgb1"
server_key: "e7cea460734e48642b7e4b42d6808f3e"
server_key_iv: "6b1e5619cff918c7597896b353964f23"
server_key_version: "encryptionkey@0"
I20260809 22:36:44.230973 30614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "958be0fda3c241378910ca8965c5cbba"
format_stamp: "Formatted at 2026-08-09 22:36:44 on dist-test-slave-rgb1"
server_key: "e7cea460734e48642b7e4b42d6808f3e"
server_key_iv: "6b1e5619cff918c7597896b353964f23"
server_key_version: "encryptionkey@0"
I20260809 22:36:44.236999 30614 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:36:44.240919 30631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:44.241868 30465 heartbeater.cc:503] Master 127.28.243.126:39609 was elected leader, sending a full tablet report...
I20260809 22:36:44.242621 30614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:36:44.242810 30614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "958be0fda3c241378910ca8965c5cbba"
format_stamp: "Formatted at 2026-08-09 22:36:44 on dist-test-slave-rgb1"
server_key: "e7cea460734e48642b7e4b42d6808f3e"
server_key_iv: "6b1e5619cff918c7597896b353964f23"
server_key_version: "encryptionkey@0"
I20260809 22:36:44.243069 30614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:44.267572 30614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:44.273556 30614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:44.274075 30614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:44.275384 30614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:36:44.277529 30614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:44.277626 30614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:44.277792 30614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:44.277870 30614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:44.327875 30614 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:36631
I20260809 22:36:44.327911 30750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:36631 every 8 connection(s)
I20260809 22:36:44.330027 30614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:36:44.332208 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 30614
I20260809 22:36:44.332484 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:36:44.333170 29645 external_mini_cluster.cc:1428] Setting key cde48e4a5964624e01546168fcaaa514
Aug 09 22:36:44 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315004, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:44.366272 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.335793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:44557 (local address 127.28.243.126:39609)
0809 22:36:44.339463 (+ 3670us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.339469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.339487 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:36:44.339546 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.339551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.339655 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.339787 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:44.342410 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.344234 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.347222 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.347777 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.347935 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.357951 (+ 10016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:44.357956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:44.357960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:44.357997 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:44.361328 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.362653 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.362659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.362663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.362810 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.363617 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.363621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.363624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.364046 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:44.364487 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.365454 (+ 967us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.365752 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3532,"thread_start_us":3396,"threads_started":1}
I20260809 22:36:44.367385 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.335029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39609 (local address 127.28.243.67:44557)
0809 22:36:44.336490 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:44.336602 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:44.338795 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:44.340120 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:44.340133 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:44.341028 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:44.342125 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.342160 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.344488 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.344496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.347024 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.347044 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.348169 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.348180 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.348597 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.354047 (+ 5450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:44.354093 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:44.357663 (+ 3570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:44.361637 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:44.361649 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:44.361671 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:44.362501 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:44.362943 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:44.362948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:44.362952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:44.363439 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:44.364453 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:44.364463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:44.364957 (+ 494us) client_negotiation.cc:769] Sending connection context
0809 22:36:44.365274 (+ 317us) client_negotiation.cc:240] Negotiation successful
0809 22:36:44.366013 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"thread_start_us":924,"threads_started":1}
I20260809 22:36:44.369387 30751 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39609
I20260809 22:36:44.369771 30751 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:44.370625 30751 heartbeater.cc:511] Master 127.28.243.126:39609 requested a full tablet report, sending...
I20260809 22:36:44.372510 30262 ts_manager.cc:194] Registered new tserver with Master: 958be0fda3c241378910ca8965c5cbba (127.28.243.67:36631)
I20260809 22:36:44.373500 30262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:44557
I20260809 22:36:44.377111 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:36:44.398948 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.385069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35888 (local address 127.28.243.126:39609)
0809 22:36:44.385286 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.385291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.385761 (+ 470us) server_negotiation.cc:407] Connection header received
0809 22:36:44.385909 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.385914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.385985 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.386078 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:44.387725 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.388525 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.389825 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.390365 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.390462 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.392994 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:44.392999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:44.393003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:44.393041 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:44.396132 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.396888 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.396892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.396895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.397079 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.397493 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.397498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.397500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.397866 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:44.398089 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.398497 (+ 408us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.398691 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 22:36:44.409905 30262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:36:44.418509 30262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:44.458655 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.437311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:36885 (local address 127.0.0.1:45900)
0809 22:36:44.448844 (+ 11533us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:44.448866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:44.449062 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:44.450559 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:44.450567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:44.450597 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:44.451051 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.451068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.453169 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.453177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.455599 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.455613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.457069 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.457075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.457461 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.457513 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 22:36:44.457822 (+ 309us) client_negotiation.cc:240] Negotiation successful
0809 22:36:44.458043 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11386,"mutex_wait_us":170,"thread_start_us":10899,"threads_started":1}
I20260809 22:36:44.459844 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.437958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.28.243.66:36885)
0809 22:36:44.445582 (+ 7624us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.445595 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.449238 (+ 3643us) server_negotiation.cc:407] Connection header received
0809 22:36:44.449490 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.449503 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.449811 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.450080 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:44.451277 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.453026 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.455766 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.456853 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.457016 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.457291 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.457864 (+ 573us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.459509 (+ 1645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7437,"thread_start_us":7307,"threads_started":1}
I20260809 22:36:44.461048 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.437068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:41391 (local address 127.0.0.1:57896)
0809 22:36:44.437907 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:44.437949 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:44.438198 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:44.451448 (+ 13250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:44.451455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:44.451482 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:44.452041 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.452056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.454340 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.454346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.457665 (+ 3319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.457680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.459847 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.459854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.460187 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.460231 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 22:36:44.460357 (+ 126us) client_negotiation.cc:240] Negotiation successful
0809 22:36:44.460503 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":347,"thread_start_us":379,"threads_started":1}
I20260809 22:36:44.461066 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.437308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.28.243.65:41391)
0809 22:36:44.450730 (+ 13422us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.450738 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.450769 (+ 31us) server_negotiation.cc:407] Connection header received
0809 22:36:44.450868 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.450885 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.451154 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.451436 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:44.452230 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.454133 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.457857 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.459603 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.459755 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.460017 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.460446 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.460684 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13233,"thread_start_us":13030,"threads_started":1}
I20260809 22:36:44.461737 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.439574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.28.243.67:36631)
0809 22:36:44.449076 (+ 9502us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.449085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.449872 (+ 787us) server_negotiation.cc:407] Connection header received
0809 22:36:44.450587 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.450595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.450815 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.450991 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:44.452084 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.454163 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.457798 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.459590 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.459756 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.460022 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.461152 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.461333 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9329,"thread_start_us":10692,"threads_started":1}
I20260809 22:36:44.461906 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.439398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36631 (local address 127.0.0.1:54990)
0809 22:36:44.449756 (+ 10358us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:44.449775 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:44.449982 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:44.451218 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:44.451222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:44.451252 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:44.451851 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.451864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.454340 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.454346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.457557 (+ 3211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.457575 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.460383 (+ 2808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.460390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.460585 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.460618 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 22:36:44.460714 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 22:36:44.460849 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10240,"mutex_wait_us":259,"thread_start_us":10097,"threads_started":1}
I20260809 22:36:44.465447 30395 tablet_service.cc:1467] Processing CreateTablet for tablet 468278a2364641a48a92d1b39cd3a7f6 (DEFAULT_TABLE table=test-table [id=f4493a98d6554851b484da1acb7c0d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:44.465622 30683 tablet_service.cc:1467] Processing CreateTablet for tablet 468278a2364641a48a92d1b39cd3a7f6 (DEFAULT_TABLE table=test-table [id=f4493a98d6554851b484da1acb7c0d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:44.465479 30539 tablet_service.cc:1467] Processing CreateTablet for tablet 468278a2364641a48a92d1b39cd3a7f6 (DEFAULT_TABLE table=test-table [id=f4493a98d6554851b484da1acb7c0d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:44.467078 30395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468278a2364641a48a92d1b39cd3a7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:44.467165 30683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468278a2364641a48a92d1b39cd3a7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:44.467211 30539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 468278a2364641a48a92d1b39cd3a7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:44.481614 30771 tablet_bootstrap.cc:492] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9: Bootstrap starting.
I20260809 22:36:44.486209 30771 tablet_bootstrap.cc:654] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:44.487042 30773 tablet_bootstrap.cc:492] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba: Bootstrap starting.
I20260809 22:36:44.487334 30772 tablet_bootstrap.cc:492] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec: Bootstrap starting.
I20260809 22:36:44.488128 30771 log.cc:824] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:44.492679 30773 tablet_bootstrap.cc:654] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:44.494128 30773 log.cc:824] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:44.494335 30772 tablet_bootstrap.cc:654] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:44.495985 30772 log.cc:824] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:44.498163 30771 tablet_bootstrap.cc:492] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9: No bootstrap required, opened a new log
I20260809 22:36:44.498296 30773 tablet_bootstrap.cc:492] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba: No bootstrap required, opened a new log
I20260809 22:36:44.498577 30771 ts_tablet_manager.cc:1397] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20260809 22:36:44.498622 30773 ts_tablet_manager.cc:1397] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260809 22:36:44.499898 30772 tablet_bootstrap.cc:492] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec: No bootstrap required, opened a new log
I20260809 22:36:44.500228 30772 ts_tablet_manager.cc:1397] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260809 22:36:44.507644 30773 raft_consensus.cc:348] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.508121 30773 raft_consensus.cc:374] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:44.508280 30773 raft_consensus.cc:729] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958be0fda3c241378910ca8965c5cbba, State: Initialized, Role: FOLLOWER
I20260809 22:36:44.509253 30773 consensus_queue.cc:260] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.510864 30751 heartbeater.cc:503] Master 127.28.243.126:39609 was elected leader, sending a full tablet report...
I20260809 22:36:44.511925 30773 ts_tablet_manager.cc:1428] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260809 22:36:44.511967 30771 raft_consensus.cc:348] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.512537 30771 raft_consensus.cc:374] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:44.512800 30771 raft_consensus.cc:729] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f50e8183c34239ace6f753ccdb6da9, State: Initialized, Role: FOLLOWER
I20260809 22:36:44.513576 30771 consensus_queue.cc:260] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.513907 30772 raft_consensus.cc:348] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.514583 30772 raft_consensus.cc:374] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:44.514765 30772 raft_consensus.cc:729] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fc8387f6024a689b2aa28e0073d6ec, State: Initialized, Role: FOLLOWER
I20260809 22:36:44.515918 30771 ts_tablet_manager.cc:1428] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260809 22:36:44.515734 30772 consensus_queue.cc:260] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.517979 30608 heartbeater.cc:503] Master 127.28.243.126:39609 was elected leader, sending a full tablet report...
I20260809 22:36:44.519066 30772 ts_tablet_manager.cc:1428] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
W20260809 22:36:44.525905 30609 tablet.cc:2369] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:36:44.583371 30752 tablet.cc:2369] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:36:44.707998 30466 tablet.cc:2369] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:36:44.849879 30779 raft_consensus.cc:482] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:44.850204 30779 raft_consensus.cc:504] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.852579 30779 leader_election.cc:288] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f50e8183c34239ace6f753ccdb6da9 (127.28.243.65:41391), 958be0fda3c241378910ca8965c5cbba (127.28.243.67:36631)
I20260809 22:36:44.860327 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.853242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:35625 (local address 127.28.243.65:41391)
0809 22:36:44.853472 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.853477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.853938 (+ 461us) server_negotiation.cc:407] Connection header received
0809 22:36:44.854042 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.854047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.854142 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.854259 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:44.855006 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.856031 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.858178 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.859279 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.859407 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.859470 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.859892 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.860008 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 22:36:44.860508 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.852926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:41391 (local address 127.28.243.66:35625)
0809 22:36:44.853682 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:44.853709 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:44.853897 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:44.854342 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:44.854348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:44.854391 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:44.854831 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.854846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.856173 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.856180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.858013 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.858025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.859500 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.859507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.859714 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.859754 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 22:36:44.859868 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 22:36:44.860021 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":24,"thread_start_us":341,"threads_started":1}
I20260809 22:36:44.861032 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.854036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:40039 (local address 127.28.243.67:36631)
0809 22:36:44.854239 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.854246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.854609 (+ 363us) server_negotiation.cc:407] Connection header received
0809 22:36:44.854776 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.854783 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.854870 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.854964 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:44.855691 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.856719 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.858838 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.859957 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.860106 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.860200 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.860659 (+ 459us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.860828 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:36:44.861776 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.853765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36631 (local address 127.28.243.66:40039)
0809 22:36:44.854505 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:44.854521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:44.854642 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:44.855000 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:44.855005 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:44.855029 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:44.855458 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.855469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.856859 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.856862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.858673 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:44.858689 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.860221 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.860227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:44.860425 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.860466 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 22:36:44.861222 (+ 756us) client_negotiation.cc:240] Negotiation successful
0809 22:36:44.861371 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":42,"thread_start_us":429,"threads_started":1}
I20260809 22:36:44.862063 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468278a2364641a48a92d1b39cd3a7f6" candidate_uuid: "68fc8387f6024a689b2aa28e0073d6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f50e8183c34239ace6f753ccdb6da9" is_pre_election: true
I20260809 22:36:44.862790 30417 raft_consensus.cc:2454] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8387f6024a689b2aa28e0073d6ec in term 0.
I20260809 22:36:44.862809 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468278a2364641a48a92d1b39cd3a7f6" candidate_uuid: "68fc8387f6024a689b2aa28e0073d6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958be0fda3c241378910ca8965c5cbba" is_pre_election: true
I20260809 22:36:44.863369 30705 raft_consensus.cc:2454] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8387f6024a689b2aa28e0073d6ec in term 0.
I20260809 22:36:44.863756 30490 leader_election.cc:302] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f50e8183c34239ace6f753ccdb6da9, 68fc8387f6024a689b2aa28e0073d6ec; no voters:
I20260809 22:36:44.864226 30779 raft_consensus.cc:2781] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:44.864451 30779 raft_consensus.cc:482] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:44.864616 30779 raft_consensus.cc:3037] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:44.867070 30779 raft_consensus.cc:504] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.867753 30779 leader_election.cc:288] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [CANDIDATE]: Term 1 election: Requested vote from peers 19f50e8183c34239ace6f753ccdb6da9 (127.28.243.65:41391), 958be0fda3c241378910ca8965c5cbba (127.28.243.67:36631)
I20260809 22:36:44.868491 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468278a2364641a48a92d1b39cd3a7f6" candidate_uuid: "68fc8387f6024a689b2aa28e0073d6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f50e8183c34239ace6f753ccdb6da9"
I20260809 22:36:44.868685 30705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "468278a2364641a48a92d1b39cd3a7f6" candidate_uuid: "68fc8387f6024a689b2aa28e0073d6ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958be0fda3c241378910ca8965c5cbba"
I20260809 22:36:44.868858 30417 raft_consensus.cc:3037] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:44.868935 30705 raft_consensus.cc:3037] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:44.871982 30417 raft_consensus.cc:2454] T 468278a2364641a48a92d1b39cd3a7f6 P 19f50e8183c34239ace6f753ccdb6da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8387f6024a689b2aa28e0073d6ec in term 1.
I20260809 22:36:44.872045 30705 raft_consensus.cc:2454] T 468278a2364641a48a92d1b39cd3a7f6 P 958be0fda3c241378910ca8965c5cbba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8387f6024a689b2aa28e0073d6ec in term 1.
I20260809 22:36:44.872637 30490 leader_election.cc:302] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fc8387f6024a689b2aa28e0073d6ec, 958be0fda3c241378910ca8965c5cbba; no voters:
I20260809 22:36:44.873176 30779 raft_consensus.cc:2781] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:44.873769 30779 raft_consensus.cc:686] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [term 1 LEADER]: Becoming Leader. State: Replica: 68fc8387f6024a689b2aa28e0073d6ec, State: Running, Role: LEADER
I20260809 22:36:44.874312 30779 consensus_queue.cc:237] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } }
I20260809 22:36:44.881875 30261 catalog_manager.cc:5495] T 468278a2364641a48a92d1b39cd3a7f6 P 68fc8387f6024a689b2aa28e0073d6ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fc8387f6024a689b2aa28e0073d6ec (127.28.243.66). New cstate: current_term: 1 leader_uuid: "68fc8387f6024a689b2aa28e0073d6ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19f50e8183c34239ace6f753ccdb6da9" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fc8387f6024a689b2aa28e0073d6ec" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 36885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "958be0fda3c241378910ca8965c5cbba" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36631 } health_report { overall_health: UNKNOWN } } }
Aug 09 22:36:44 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:36:44.930961 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.913940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.28.243.65:41391)
0809 22:36:44.914234 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.914241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.914641 (+ 400us) server_negotiation.cc:407] Connection header received
0809 22:36:44.914784 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.914788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.914884 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.915005 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:44.916674 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.917608 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.919008 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.919607 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.919729 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.924424 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:44.924439 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:44.924456 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:44.924496 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:44.927763 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.928490 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.928499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.928501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.928728 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.929218 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.929223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.929225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.929710 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:44.929972 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.930265 (+ 293us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.930549 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Aug 09 22:36:44 dist-test-slave-rgb1 krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 22:36:44 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:36:44.990176 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:44.973627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.28.243.65:41391)
0809 22:36:44.973895 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:44.973904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:44.974293 (+ 389us) server_negotiation.cc:407] Connection header received
0809 22:36:44.974504 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:44.974510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:44.974585 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:44.974695 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:44.976307 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.977236 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.978521 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:44.979031 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:44.979170 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:44.984016 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:44.984023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:44.984027 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:44.984074 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:44.987260 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.988040 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.988048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.988050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.988193 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:44.988695 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:44.988699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:44.988701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:44.989004 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:44.989242 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:44.989666 (+ 424us) server_negotiation.cc:299] Negotiation successful
0809 22:36:44.989927 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Aug 09 22:36:45 dist-test-slave-rgb1 krb5kdc[30214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 22:36:45 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:36:45.046322 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:45.031355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.28.243.65:41391)
0809 22:36:45.031569 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:45.031576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:45.031984 (+ 408us) server_negotiation.cc:407] Connection header received
0809 22:36:45.032227 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:45.032232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:45.032302 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:45.032413 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:45.033928 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.034637 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.035881 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.036328 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.036468 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:45.040728 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:45.040733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:45.040736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:45.040774 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:45.043459 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.044204 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.044213 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.044216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.044351 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.044839 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.044845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.044849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.045130 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:45.045400 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:45.045692 (+ 292us) server_negotiation.cc:299] Negotiation successful
0809 22:36:45.045931 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 22:36:45.048708 30373 tablet_service.cc:2908] Rejecting scan request for tablet 468278a2364641a48a92d1b39cd3a7f6: Uninitialized: clean time has not yet been initialized
I20260809 22:36:45.052443 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30327
I20260809 22:36:45.068450 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:45.053696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.28.243.65:41391)
0809 22:36:45.053934 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:45.053940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:45.054398 (+ 458us) server_negotiation.cc:407] Connection header received
0809 22:36:45.054591 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:45.054600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:45.054701 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:45.054877 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:45.056169 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.057191 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.058535 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.059086 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.059213 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:45.062033 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:45.062039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:45.062042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:45.062080 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:45.065296 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.066083 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.066088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.066090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.066223 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.066788 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.066793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.066795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.067220 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:45.067498 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:45.067796 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 22:36:45.068015 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 22:36:45.069325 30460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:45.166415 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 30327
I20260809 22:36:45.186589 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30471
Aug 09 22:36:45 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:36:45.204227 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:45.187389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.28.243.66:36885)
0809 22:36:45.187949 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:45.187954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:45.187972 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:36:45.188028 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:45.188031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:45.188099 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:45.188192 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:45.189708 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.190555 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.192120 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.192692 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.192812 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:45.197418 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:45.197438 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:45.197455 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:45.197513 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:45.200886 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.201665 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.201673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.201675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.201835 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.202302 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.202306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.202308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.202773 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:45.203053 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:45.203519 (+ 466us) server_negotiation.cc:299] Negotiation successful
0809 22:36:45.203757 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":314,"threads_started":1}
I20260809 22:36:45.205003 30602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:45.301481 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 30471
I20260809 22:36:45.322259 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30614
Aug 09 22:36:45 dist-test-slave-rgb1 krb5kdc[30214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:36:45.339393 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:45.323291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.28.243.67:36631)
0809 22:36:45.323541 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:45.323547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:45.323569 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:36:45.323743 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:45.323748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:45.323828 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:45.323937 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:45.325354 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.326118 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.327541 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:45.327968 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:45.328102 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:45.332886 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:45.332899 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:45.332914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:45.332949 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:45.336355 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.337184 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.337196 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.337198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.337335 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:45.337766 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:45.337770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:45.337772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:45.338165 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:45.338484 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:45.338778 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 22:36:45.339041 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 22:36:45.340215 30746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:45.445612 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 30614
I20260809 22:36:45.465032 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30227
I20260809 22:36:45.466617 30294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:45.578770 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 30227
2026-08-09T22:36:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5816 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:36:45 dist-test-slave-rgb1 krb5kdc[30805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:36:45 dist-test-slave-rgb1 krb5kdc[30805](info): set up 2 sockets
Aug 09 22:36:45 dist-test-slave-rgb1 krb5kdc[30805](info): commencing operation
krb5kdc: starting...
W20260809 22:36:46.783740 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:36:46 dist-test-slave-rgb1 krb5kdc[30805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:36:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:36:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:36:47.017827 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:45337
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:47.363369 30818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:47.363680 30818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:47.363834 30818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:47.371336 30818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:36:47.371449 30818 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 22:36:47.371536 30818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:47.371622 30818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:47.371742 30818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:36:47.371852 30818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:36:47.382035 30818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:45337
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:47.383442 30818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:47.384833 30818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:47.400807 30825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:47.400758 30827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:47.401690 30829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:47.402613 30818 server_base.cc:1034] running on GCE node
I20260809 22:36:47.403738 30818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:47.406028 30818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:47.407322 30818 hybrid_clock.cc:648] HybridClock initialized: now 1786315007407248 us; error 39 us; skew 500 ppm
I20260809 22:36:47.407711 30818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:47 dist-test-slave-rgb1 krb5kdc[30805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:47.411270 30818 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:36:47.412968 30818 webserver.cc:460] Webserver started at http://127.28.243.126:38493/ using document root <none> and password file <none>
I20260809 22:36:47.413724 30818 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:47.413864 30818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:47.414211 30818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:47.417407 30818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36"
format_stamp: "Formatted at 2026-08-09 22:36:47 on dist-test-slave-rgb1"
server_key: "90c8289916c5a971283590ef49cb6de1"
server_key_iv: "ad8d5d881f656d996aa5311064e90c94"
server_key_version: "encryptionkey@0"
I20260809 22:36:47.418267 30818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36"
format_stamp: "Formatted at 2026-08-09 22:36:47 on dist-test-slave-rgb1"
server_key: "90c8289916c5a971283590ef49cb6de1"
server_key_iv: "ad8d5d881f656d996aa5311064e90c94"
server_key_version: "encryptionkey@0"
I20260809 22:36:47.423996 30818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 22:36:47.428048 30837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:47.429662 30818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:36:47.429874 30818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36"
format_stamp: "Formatted at 2026-08-09 22:36:47 on dist-test-slave-rgb1"
server_key: "90c8289916c5a971283590ef49cb6de1"
server_key_iv: "ad8d5d881f656d996aa5311064e90c94"
server_key_version: "encryptionkey@0"
I20260809 22:36:47.430125 30818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:47.460408 30818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:47.465135 30818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:47.465431 30818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:47.490952 30818 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:45337
I20260809 22:36:47.490998 30900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:45337 every 8 connection(s)
I20260809 22:36:47.492450 30818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:36:47.496680 30901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:47.501312 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 30818
I20260809 22:36:47.501623 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:36:47.502338 29645 external_mini_cluster.cc:1428] Setting key bae202b33cef835b021fbac563e147cb
I20260809 22:36:47.509260 30901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36: Bootstrap starting.
I20260809 22:36:47.513188 30901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:47.514266 30901 log.cc:824] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:47.517373 30901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36: No bootstrap required, opened a new log
Aug 09 22:36:47 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:47.528297 30901 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45337 } }
I20260809 22:36:47.528815 30901 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:47.528930 30901 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d8d3071c9c42bfb73c02c9d7a53b36, State: Initialized, Role: FOLLOWER
I20260809 22:36:47.529659 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:47.504915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.28.243.126:45337)
0809 22:36:47.505852 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:47.505869 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:47.505931 (+ 62us) server_negotiation.cc:407] Connection header received
0809 22:36:47.508377 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:47.508407 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:47.508973 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:47.509432 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:47.511041 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:47.513092 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:47.514597 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:47.515487 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:47.515682 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:47.520066 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:47.520080 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:47.520100 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:47.520147 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:47.524669 (+ 4522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:47.525438 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:47.525447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:47.525456 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:47.525714 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:47.526133 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:47.526138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:47.526142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:47.526963 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:47.527278 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:47.527692 (+ 414us) server_negotiation.cc:299] Negotiation successful
0809 22:36:47.528150 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":406,"threads_started":1}
I20260809 22:36:47.529636 30901 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45337 } }
I20260809 22:36:47.529968 30901 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:36:47.530130 30901 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:36:47.530328 30901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:47.533154 30901 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45337 } }
I20260809 22:36:47.533694 30901 leader_election.cc:302] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d8d3071c9c42bfb73c02c9d7a53b36; no voters:
I20260809 22:36:47.534344 30901 leader_election.cc:288] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:36:47.534504 30906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:47.535682 30906 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [term 1 LEADER]: Becoming Leader. State: Replica: b2d8d3071c9c42bfb73c02c9d7a53b36, State: Running, Role: LEADER
I20260809 22:36:47.536263 30906 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45337 } }
I20260809 22:36:47.536686 30901 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:36:47.540244 30908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d8d3071c9c42bfb73c02c9d7a53b36. Latest consensus state: current_term: 1 leader_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45337 } } }
I20260809 22:36:47.540319 30907 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d8d3071c9c42bfb73c02c9d7a53b36" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45337 } } }
I20260809 22:36:47.540766 30908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:47.540850 30907 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36 [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:47.541837 30912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:36:47.548808 30912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:36:47.558641 30923 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:36:47.560328 30912 catalog_manager.cc:1347] Generated new cluster ID: 624abad1036e4bb7a164e60db24ed7bc
I20260809 22:36:47.560446 30912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:36:47.576030 30912 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:36:47.577993 30912 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:36:47.589295 30912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36: Generated new TSK 0
I20260809 22:36:47.590431 30912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:36:47.649384 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45337
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:48.012848 30927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:48.013178 30927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:48.013355 30927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:48.020802 30927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:48.020928 30927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:48.021087 30927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:36:48.021216 30927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:36:48.031503 30927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:48.033038 30927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:48.034507 30927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:48.048240 30932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:48.048494 30933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:48.048614 30936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:48.049927 30927 server_base.cc:1034] running on GCE node
I20260809 22:36:48.050738 30927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:48.051944 30927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:48.053226 30927 hybrid_clock.cc:648] HybridClock initialized: now 1786315008053114 us; error 83 us; skew 500 ppm
I20260809 22:36:48.053575 30927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:48 dist-test-slave-rgb1 krb5kdc[30805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315008, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:48.057235 30927 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:36:48.058954 30927 webserver.cc:460] Webserver started at http://127.28.243.65:39881/ using document root <none> and password file <none>
I20260809 22:36:48.059774 30927 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:48.059870 30927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:48.060164 30927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:48.063376 30927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "56601b7d9544443aab8fa90dc3cbcc1f"
format_stamp: "Formatted at 2026-08-09 22:36:48 on dist-test-slave-rgb1"
server_key: "618d4dd47202d2828375da0b5c0ea516"
server_key_iv: "eabde8c64ce24dece84f18e12537c9c8"
server_key_version: "encryptionkey@0"
I20260809 22:36:48.064179 30927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "56601b7d9544443aab8fa90dc3cbcc1f"
format_stamp: "Formatted at 2026-08-09 22:36:48 on dist-test-slave-rgb1"
server_key: "618d4dd47202d2828375da0b5c0ea516"
server_key_iv: "eabde8c64ce24dece84f18e12537c9c8"
server_key_version: "encryptionkey@0"
I20260809 22:36:48.070192 30927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 22:36:48.074450 30943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:48.076418 30927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260809 22:36:48.076691 30927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "56601b7d9544443aab8fa90dc3cbcc1f"
format_stamp: "Formatted at 2026-08-09 22:36:48 on dist-test-slave-rgb1"
server_key: "618d4dd47202d2828375da0b5c0ea516"
server_key_iv: "eabde8c64ce24dece84f18e12537c9c8"
server_key_version: "encryptionkey@0"
I20260809 22:36:48.076974 30927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:48.099675 30927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:48.104928 30927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:48.105268 30927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:48.108450 30927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:48.108543 30927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:48.108752 30927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:48.108865 30927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:36:48 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315008, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:48.135568 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.109121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:60337 (local address 127.28.243.126:45337)
0809 22:36:48.109842 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:48.109859 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:48.110903 (+ 1044us) server_negotiation.cc:407] Connection header received
0809 22:36:48.113104 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:48.113111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:48.113220 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:48.113409 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:48.116378 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.118069 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.120970 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.121525 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.121878 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.127014 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:48.127020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:48.127024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:48.127065 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:48.130914 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.132275 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.132287 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.132294 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.132537 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.133465 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.133471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.133474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.133819 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:48.133985 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:48.134903 (+ 918us) server_negotiation.cc:299] Negotiation successful
0809 22:36:48.135177 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":475,"threads_started":1}
I20260809 22:36:48.136956 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.109755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45337 (local address 127.28.243.65:60337)
0809 22:36:48.110694 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:48.110745 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:48.112789 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:48.113730 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:48.113746 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:48.115014 (+ 1268us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:48.116084 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.116128 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.118310 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.118317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.120779 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.120799 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.122058 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.122070 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.122351 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.123348 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:48.123387 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:48.126690 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:48.131159 (+ 4469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.131167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.131183 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.132061 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.132801 (+ 740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.132812 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.132817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.133299 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.133968 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:48.133977 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:48.134489 (+ 512us) client_negotiation.cc:769] Sending connection context
0809 22:36:48.134788 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 22:36:48.135373 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":415,"threads_started":1}
I20260809 22:36:48.167220 30927 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:38287
I20260809 22:36:48.167245 31069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:38287 every 8 connection(s)
I20260809 22:36:48.167830 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.148866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:44037 (local address 127.28.243.126:45337)
0809 22:36:48.149206 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:48.149216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:48.149997 (+ 781us) server_negotiation.cc:407] Connection header received
0809 22:36:48.150368 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:48.150377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:48.150493 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:48.150612 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:48.152364 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.153500 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.156282 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.156873 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.157037 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.160413 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:48.160419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:48.160423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:48.160463 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:48.164731 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.165535 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.165542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.165545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.165692 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.166166 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.166174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.166178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.166564 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:48.166736 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:48.167181 (+ 445us) server_negotiation.cc:299] Negotiation successful
0809 22:36:48.167444 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260809 22:36:48.167982 31026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.148937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45337 (local address 127.28.243.65:44037)
0809 22:36:48.149833 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:48.149864 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:48.150120 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:48.150779 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:48.150783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:48.151503 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:48.152187 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.152205 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.153655 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.153661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.156089 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.156108 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.157120 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.157129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.157341 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.158419 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:48.158458 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:48.160145 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:48.164913 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.164919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.164924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.165390 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.165819 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.165824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.165828 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.165983 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.166702 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:48.166707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:48.166910 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 22:36:48.167078 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 22:36:48.167506 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":387,"threads_started":1}
I20260809 22:36:48.169083 30927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:36:48.172765 31070 heartbeater.cc:347] Connected to a master server at 127.28.243.126:45337
I20260809 22:36:48.173094 31070 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:48.173864 31070 heartbeater.cc:511] Master 127.28.243.126:45337 requested a full tablet report, sending...
I20260809 22:36:48.174661 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 30927
I20260809 22:36:48.174937 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:36:48.175668 29645 external_mini_cluster.cc:1428] Setting key 4ba767fe5828f8a8a95ff02176248f3c
I20260809 22:36:48.176141 30854 ts_manager.cc:194] Registered new tserver with Master: 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65:38287)
I20260809 22:36:48.178304 30854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:60337
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:36:48.230093 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45337
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:48.582551 31074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:48.582924 31074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:48.583101 31074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:48.590662 31074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:48.590835 31074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:48.591064 31074 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:36:48.591313 31074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:36:48.603363 31074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:48.604982 31074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:48.606493 31074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:48.621356 31084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:48.621486 31082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:48.621438 31081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:48.622704 31074 server_base.cc:1034] running on GCE node
I20260809 22:36:48.623452 31074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:48.624486 31074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:48.625783 31074 hybrid_clock.cc:648] HybridClock initialized: now 1786315008625670 us; error 90 us; skew 500 ppm
I20260809 22:36:48.626180 31074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:48 dist-test-slave-rgb1 krb5kdc[30805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315008, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:48.629853 31074 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:36:48.631652 31074 webserver.cc:460] Webserver started at http://127.28.243.66:35525/ using document root <none> and password file <none>
I20260809 22:36:48.632406 31074 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:48.632542 31074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:48.632956 31074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:48.636189 31074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "a2f2cef657584b66b306c3c17ab02d69"
format_stamp: "Formatted at 2026-08-09 22:36:48 on dist-test-slave-rgb1"
server_key: "7ffae95534a95622b7d092b69526ace7"
server_key_iv: "60a4e7f2f77cac9d2b6ef728626da2e0"
server_key_version: "encryptionkey@0"
I20260809 22:36:48.637136 31074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "a2f2cef657584b66b306c3c17ab02d69"
format_stamp: "Formatted at 2026-08-09 22:36:48 on dist-test-slave-rgb1"
server_key: "7ffae95534a95622b7d092b69526ace7"
server_key_iv: "60a4e7f2f77cac9d2b6ef728626da2e0"
server_key_version: "encryptionkey@0"
I20260809 22:36:48.643047 31074 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260809 22:36:48.646956 31091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:48.648536 31074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:36:48.648836 31074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "a2f2cef657584b66b306c3c17ab02d69"
format_stamp: "Formatted at 2026-08-09 22:36:48 on dist-test-slave-rgb1"
server_key: "7ffae95534a95622b7d092b69526ace7"
server_key_iv: "60a4e7f2f77cac9d2b6ef728626da2e0"
server_key_version: "encryptionkey@0"
I20260809 22:36:48.649209 31074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:48.677241 31074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:48.682456 31074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:48.682758 31074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:48.685854 31074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:48.685971 31074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:48.686131 31074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:48.686188 31074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:36:48 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315008, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:48.713824 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.686843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:57431 (local address 127.28.243.126:45337)
0809 22:36:48.687348 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:48.687357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:48.688983 (+ 1626us) server_negotiation.cc:407] Connection header received
0809 22:36:48.691981 (+ 2998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:48.691988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:48.692095 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:48.692228 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:48.694950 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.696340 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.699454 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.699959 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.700111 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.705274 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:48.705285 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:48.705293 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:48.705355 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:48.709599 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.710884 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.710893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.710897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.711074 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.711623 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.711634 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.711636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.711955 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:48.712153 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:48.713090 (+ 937us) server_negotiation.cc:299] Negotiation successful
0809 22:36:48.713356 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":241,"threads_started":1}
I20260809 22:36:48.715173 31104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.687609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45337 (local address 127.28.243.66:57431)
0809 22:36:48.688741 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:48.688797 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:48.691577 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:48.692519 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:48.692529 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:48.693511 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:48.694634 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.694661 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.696593 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.696604 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.699210 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.699225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.700226 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.700233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.700467 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.701308 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:48.701354 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:48.704889 (+ 3535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:48.709844 (+ 4955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.709853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.709872 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.710638 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.711219 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.711225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.711229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.711481 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.712102 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:48.712114 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:48.712610 (+ 496us) client_negotiation.cc:769] Sending connection context
0809 22:36:48.712973 (+ 363us) client_negotiation.cc:240] Negotiation successful
0809 22:36:48.713587 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":444,"threads_started":1}
I20260809 22:36:48.738574 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.723034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45337 (local address 127.28.243.66:36537)
0809 22:36:48.723704 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:48.723722 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:48.723997 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:48.724384 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:48.724389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:48.725082 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:48.725430 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.725441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.726501 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.726507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.727597 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:48.727609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.728419 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.728426 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:48.728686 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.729494 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:48.729513 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:48.730504 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:48.735317 (+ 4813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.735325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.735329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.735803 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.736293 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:48.736299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:48.736303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:48.736460 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:48.737278 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:48.737283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:48.737523 (+ 240us) client_negotiation.cc:769] Sending connection context
0809 22:36:48.737721 (+ 198us) client_negotiation.cc:240] Negotiation successful
0809 22:36:48.738166 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":334,"threads_started":1}
I20260809 22:36:48.739109 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:48.722939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:36537 (local address 127.28.243.126:45337)
0809 22:36:48.723119 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:48.723124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:48.723817 (+ 693us) server_negotiation.cc:407] Connection header received
0809 22:36:48.724155 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:48.724161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:48.724241 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:48.724363 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:48.725594 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.726346 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.727764 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:48.728207 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:48.728374 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:48.730718 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:48.730724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:48.730728 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:48.730762 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:48.735153 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.735998 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.736005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.736009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.736156 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:48.736614 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:48.736620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:48.736624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:48.737135 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:48.737311 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:48.737873 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 22:36:48.738212 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 22:36:48.746802 31074 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:43441
I20260809 22:36:48.746836 31223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:43441 every 8 connection(s)
I20260809 22:36:48.748029 31074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:36:48.750530 31224 heartbeater.cc:347] Connected to a master server at 127.28.243.126:45337
I20260809 22:36:48.750819 31224 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:48.751818 31224 heartbeater.cc:511] Master 127.28.243.126:45337 requested a full tablet report, sending...
I20260809 22:36:48.753484 30854 ts_manager.cc:194] Registered new tserver with Master: a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66:43441)
I20260809 22:36:48.754565 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 31074
I20260809 22:36:48.754699 30854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:57431
I20260809 22:36:48.754803 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:36:48.755326 29645 external_mini_cluster.cc:1428] Setting key 55d0c37f1e837c089dfab89cbf0c86cd
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:36:48.809082 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45337
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:49.163955 31229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:49.164304 31229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:49.164480 31229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:49.171998 31229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:49.172139 31229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:49.172290 31229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:36:49.172429 31229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:36:49.181722 31070 heartbeater.cc:503] Master 127.28.243.126:45337 was elected leader, sending a full tablet report...
I20260809 22:36:49.184316 31229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:45337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:49.185794 31229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:49.187222 31229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:49.201622 31236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:49.201655 31238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:49.201514 31235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:49.202946 31229 server_base.cc:1034] running on GCE node
I20260809 22:36:49.203686 31229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:49.204829 31229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:49.206107 31229 hybrid_clock.cc:648] HybridClock initialized: now 1786315009205979 us; error 80 us; skew 500 ppm
I20260809 22:36:49.206434 31229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:49 dist-test-slave-rgb1 krb5kdc[30805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315009, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:49.210294 31229 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:36:49.211864 31229 webserver.cc:460] Webserver started at http://127.28.243.67:42901/ using document root <none> and password file <none>
I20260809 22:36:49.212644 31229 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:49.212762 31229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:49.213107 31229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:49.216279 31229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "aedac66ab5bf4a2fa628c1bf1e894d71"
format_stamp: "Formatted at 2026-08-09 22:36:49 on dist-test-slave-rgb1"
server_key: "166f3a5cf6968d40661e71424c056122"
server_key_iv: "937ae1bdb5ed06e4d399b1cad5044149"
server_key_version: "encryptionkey@0"
I20260809 22:36:49.217149 31229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "aedac66ab5bf4a2fa628c1bf1e894d71"
format_stamp: "Formatted at 2026-08-09 22:36:49 on dist-test-slave-rgb1"
server_key: "166f3a5cf6968d40661e71424c056122"
server_key_iv: "937ae1bdb5ed06e4d399b1cad5044149"
server_key_version: "encryptionkey@0"
I20260809 22:36:49.222622 31229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 22:36:49.226372 31246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:49.227803 31229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 22:36:49.227979 31229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "aedac66ab5bf4a2fa628c1bf1e894d71"
format_stamp: "Formatted at 2026-08-09 22:36:49 on dist-test-slave-rgb1"
server_key: "166f3a5cf6968d40661e71424c056122"
server_key_iv: "937ae1bdb5ed06e4d399b1cad5044149"
server_key_version: "encryptionkey@0"
I20260809 22:36:49.228186 31229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:49.242452 31229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:49.246944 31229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:49.247241 31229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:49.250183 31229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:49.250284 31229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:49.250442 31229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:49.250537 31229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:36:49 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315009, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:49.278710 31258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.251410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45337 (local address 127.28.243.67:38291)
0809 22:36:49.252298 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.252337 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.254803 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.255931 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.255948 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.256966 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:49.258118 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.258155 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.259735 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.259743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.261914 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.261929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.263121 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.263129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.263461 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.264442 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:49.264489 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:49.268477 (+ 3988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:49.272740 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:49.272756 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:49.272780 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:49.273708 (+ 928us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:49.274292 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:49.274304 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:49.274307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:49.274672 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:49.275490 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:49.275499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:49.276333 (+ 834us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.276819 (+ 486us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.277482 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":408,"threads_started":1}
I20260809 22:36:49.278983 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.250837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:38291 (local address 127.28.243.126:45337)
0809 22:36:49.251519 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.251524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.252493 (+ 969us) server_negotiation.cc:407] Connection header received
0809 22:36:49.255166 (+ 2673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.255173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.255280 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.255436 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:49.258366 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.259496 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.262099 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.262820 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.262986 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.268909 (+ 5923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:49.268920 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:49.268926 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:49.269004 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:49.272501 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.273932 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.273941 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.273946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.274145 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.274820 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.274826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.274829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.275306 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:49.275502 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.277988 (+ 2486us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.278411 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":429,"threads_started":1}
I20260809 22:36:49.300350 31229 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:34293
I20260809 22:36:49.300431 31375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:34293 every 8 connection(s)
I20260809 22:36:49.301622 31229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:36:49.303081 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 31229
I20260809 22:36:49.303321 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:36:49.303889 29645 external_mini_cluster.cc:1428] Setting key 3c451076dcbca76a4c345b68662f4b08
I20260809 22:36:49.305012 31376 heartbeater.cc:347] Connected to a master server at 127.28.243.126:45337
I20260809 22:36:49.305269 31376 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:49.306097 31376 heartbeater.cc:511] Master 127.28.243.126:45337 requested a full tablet report, sending...
I20260809 22:36:49.307682 30854 ts_manager.cc:194] Registered new tserver with Master: aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67:34293)
I20260809 22:36:49.307723 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.292054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:44871 (local address 127.28.243.126:45337)
0809 22:36:49.292307 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.292317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.293020 (+ 703us) server_negotiation.cc:407] Connection header received
0809 22:36:49.293259 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.293264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.293346 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.293477 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:49.294948 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.295738 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.297894 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.298428 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.298561 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.301133 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:49.301138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:49.301141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:49.301173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:49.304546 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.305472 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.305477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.305481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.305648 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.306132 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.306137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.306140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.306543 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:49.306728 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.307226 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.307459 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 22:36:49.308008 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.292037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45337 (local address 127.28.243.67:44871)
0809 22:36:49.292860 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.292887 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.293094 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.293519 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.293524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.294270 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:49.294779 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.294795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.295886 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.295894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.297730 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.297743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.298630 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.298638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.298816 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.299533 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:49.299565 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:49.300932 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:49.304714 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:49.304720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:49.304725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:49.305319 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:49.305781 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:49.305786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:49.305789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:49.305975 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:49.306688 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:49.306695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:49.306926 (+ 231us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.307077 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.307437 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":410,"threads_started":1}
I20260809 22:36:49.308717 30854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:38291
I20260809 22:36:49.319996 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 22:36:49 dist-test-slave-rgb1 krb5kdc[30805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 22:36:49 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:49.385159 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.360149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.28.243.126:45337)
0809 22:36:49.360440 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.360447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.360778 (+ 331us) server_negotiation.cc:407] Connection header received
0809 22:36:49.360912 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.360916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.361037 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.361145 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:49.362534 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.363369 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.364765 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.365270 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.365405 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.379318 (+ 13913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:49.379325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:49.379328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:49.379375 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:49.382542 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.383283 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.383287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.383289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.383409 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.383747 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.383751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.383753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.384058 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:49.384265 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.384535 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.384755 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 22:36:49.389607 30853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:36:49.397147 30853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:49.426105 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.412142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.28.243.66:43441)
0809 22:36:49.413914 (+ 1772us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.413923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.417289 (+ 3366us) server_negotiation.cc:407] Connection header received
0809 22:36:49.417653 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.417667 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.418007 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.418267 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:49.419067 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.420819 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.423496 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.424637 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.424806 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.425056 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.425531 (+ 475us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.425737 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1615,"thread_start_us":1482,"threads_started":1}
I20260809 22:36:49.426198 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.411953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:43441 (local address 127.0.0.1:60190)
0809 22:36:49.417106 (+ 5153us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.417157 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.417403 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.418271 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.418279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.418318 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:49.418867 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.418881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.420998 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.421014 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.423311 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.423325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.424888 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.424895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.425221 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.425286 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.425461 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.425695 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4775,"mutex_wait_us":41,"thread_start_us":375,"threads_started":1}
I20260809 22:36:49.428465 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.413088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47322 (local address 127.28.243.65:38287)
0809 22:36:49.418065 (+ 4977us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.418076 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.418106 (+ 30us) server_negotiation.cc:407] Connection header received
0809 22:36:49.419256 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.419267 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.419518 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.419735 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:49.420359 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.421881 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.425321 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.427388 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.427565 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.427795 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.428065 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.428225 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4799,"thread_start_us":4665,"threads_started":1}
I20260809 22:36:49.428550 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.412332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38287 (local address 127.0.0.1:47322)
0809 22:36:49.417106 (+ 4774us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.417157 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.417403 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.419773 (+ 2370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.419778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.419809 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:49.420190 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.420200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.422102 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.422108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.425116 (+ 3008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.425130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.427628 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.427635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.427846 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.427882 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.427997 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.428154 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4614,"mutex_wait_us":86,"thread_start_us":4392,"threads_started":1}
I20260809 22:36:49.430938 31147 tablet_service.cc:1467] Processing CreateTablet for tablet 382654e1ecd949b091265bd1a44a15c2 (DEFAULT_TABLE table=test-table [id=f91c4146fbeb4974bba1138a2b2d36d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:49.432435 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382654e1ecd949b091265bd1a44a15c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.432054 30996 tablet_service.cc:1467] Processing CreateTablet for tablet 382654e1ecd949b091265bd1a44a15c2 (DEFAULT_TABLE table=test-table [id=f91c4146fbeb4974bba1138a2b2d36d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:49.433367 30996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382654e1ecd949b091265bd1a44a15c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.433744 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.414450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:34293 (local address 127.0.0.1:46976)
0809 22:36:49.418556 (+ 4106us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.418577 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.418748 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.419373 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.419378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.419407 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:49.419785 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.419798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.422170 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.422176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.426108 (+ 3932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.426133 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.432776 (+ 6643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.432784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.432988 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.433036 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.433149 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.433306 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3914,"mutex_wait_us":3194,"thread_start_us":1097,"threads_started":1}
I20260809 22:36:49.434757 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.414084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.28.243.67:34293)
0809 22:36:49.414836 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.414842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.418787 (+ 3945us) server_negotiation.cc:407] Connection header received
0809 22:36:49.418914 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.418922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.419123 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.419367 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:49.420047 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.421974 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.426374 (+ 4400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.427430 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.427600 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.434038 (+ 6438us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.434174 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.434373 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":390,"threads_started":1}
I20260809 22:36:49.438934 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 382654e1ecd949b091265bd1a44a15c2 (DEFAULT_TABLE table=test-table [id=f91c4146fbeb4974bba1138a2b2d36d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:49.440666 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 382654e1ecd949b091265bd1a44a15c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.446341 31395 tablet_bootstrap.cc:492] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Bootstrap starting.
I20260809 22:36:49.448894 31396 tablet_bootstrap.cc:492] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Bootstrap starting.
I20260809 22:36:49.450904 31395 tablet_bootstrap.cc:654] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.452319 31395 log.cc:824] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:49.452772 31396 tablet_bootstrap.cc:654] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.454125 31396 log.cc:824] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:49.455796 31395 tablet_bootstrap.cc:492] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: No bootstrap required, opened a new log
I20260809 22:36:49.456138 31395 ts_tablet_manager.cc:1397] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 22:36:49.457362 31396 tablet_bootstrap.cc:492] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: No bootstrap required, opened a new log
I20260809 22:36:49.457662 31396 ts_tablet_manager.cc:1397] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260809 22:36:49.458817 31399 tablet_bootstrap.cc:492] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Bootstrap starting.
I20260809 22:36:49.462357 31399 tablet_bootstrap.cc:654] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.463430 31399 log.cc:824] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:49.466532 31399 tablet_bootstrap.cc:492] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: No bootstrap required, opened a new log
I20260809 22:36:49.466830 31399 ts_tablet_manager.cc:1397] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 22:36:49.467829 31395 raft_consensus.cc:348] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.468331 31395 raft_consensus.cc:374] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.468442 31395 raft_consensus.cc:729] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56601b7d9544443aab8fa90dc3cbcc1f, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.469033 31396 raft_consensus.cc:348] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.469458 31396 raft_consensus.cc:374] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.469290 31395 consensus_queue.cc:260] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.469589 31396 raft_consensus.cc:729] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f2cef657584b66b306c3c17ab02d69, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.470309 31396 consensus_queue.cc:260] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.471390 31395 ts_tablet_manager.cc:1428] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260809 22:36:49.472595 31224 heartbeater.cc:503] Master 127.28.243.126:45337 was elected leader, sending a full tablet report...
I20260809 22:36:49.473117 31396 ts_tablet_manager.cc:1428] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260809 22:36:49.479591 31399 raft_consensus.cc:348] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.480126 31399 raft_consensus.cc:374] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.480252 31399 raft_consensus.cc:729] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedac66ab5bf4a2fa628c1bf1e894d71, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.481029 31399 consensus_queue.cc:260] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.482442 31376 heartbeater.cc:503] Master 127.28.243.126:45337 was elected leader, sending a full tablet report...
I20260809 22:36:49.482949 31399 ts_tablet_manager.cc:1428] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260809 22:36:49.499567 31402 raft_consensus.cc:482] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:49.499958 31402 raft_consensus.cc:504] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
W20260809 22:36:49.500321 31226 tablet.cc:2369] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:36:49.503561 31402 leader_election.cc:288] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65:38287), aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67:34293)
I20260809 22:36:49.510576 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.503371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:35773 (local address 127.28.243.67:34293)
0809 22:36:49.503647 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.503657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.503684 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:36:49.503768 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.503774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.503870 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.503993 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:49.504947 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.506050 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.508355 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.509652 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.509862 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.509955 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.510245 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.510361 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260809 22:36:49.510984 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.502446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:60227 (local address 127.28.243.65:38287)
0809 22:36:49.502828 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.502839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.504313 (+ 1474us) server_negotiation.cc:407] Connection header received
0809 22:36:49.504512 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.504518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.504659 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.504773 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:49.505387 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.506417 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.508620 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.510002 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.510147 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.510237 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.510622 (+ 385us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.510761 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260809 22:36:49.511107 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.502401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38287 (local address 127.28.243.66:60227)
0809 22:36:49.504191 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.504209 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.504341 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.504793 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.504798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.504824 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:49.505223 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.505234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.506571 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.506577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.508419 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.508437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.510254 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.510261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.510431 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.510466 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.510578 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.510740 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1646,"thread_start_us":1483,"threads_started":1}
I20260809 22:36:49.511198 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.502245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:34293 (local address 127.28.243.66:35773)
0809 22:36:49.503035 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.503059 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.503194 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.504194 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.504197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.504222 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:49.504742 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.504761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.506204 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.506210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.508184 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.508197 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.509879 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.509884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.510028 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.510064 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.510211 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.510371 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"spinlock_wait_cycles":4864,"thread_start_us":1038,"threads_started":1}
I20260809 22:36:49.512450 31022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382654e1ecd949b091265bd1a44a15c2" candidate_uuid: "a2f2cef657584b66b306c3c17ab02d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" is_pre_election: true
I20260809 22:36:49.512459 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382654e1ecd949b091265bd1a44a15c2" candidate_uuid: "a2f2cef657584b66b306c3c17ab02d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" is_pre_election: true
I20260809 22:36:49.513173 31022 raft_consensus.cc:2454] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f2cef657584b66b306c3c17ab02d69 in term 0.
I20260809 22:36:49.513173 31323 raft_consensus.cc:2454] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2f2cef657584b66b306c3c17ab02d69 in term 0.
I20260809 22:36:49.514091 31094 leader_election.cc:302] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56601b7d9544443aab8fa90dc3cbcc1f, a2f2cef657584b66b306c3c17ab02d69, aedac66ab5bf4a2fa628c1bf1e894d71; no voters:
I20260809 22:36:49.514719 31402 raft_consensus.cc:2781] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:49.514951 31402 raft_consensus.cc:482] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:49.515076 31402 raft_consensus.cc:3037] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:49.517606 31402 raft_consensus.cc:504] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.518308 31402 leader_election.cc:288] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [CANDIDATE]: Term 1 election: Requested vote from peers 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65:38287), aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67:34293)
I20260809 22:36:49.518864 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382654e1ecd949b091265bd1a44a15c2" candidate_uuid: "a2f2cef657584b66b306c3c17ab02d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71"
I20260809 22:36:49.519081 31323 raft_consensus.cc:3037] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:49.520805 31022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "382654e1ecd949b091265bd1a44a15c2" candidate_uuid: "a2f2cef657584b66b306c3c17ab02d69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56601b7d9544443aab8fa90dc3cbcc1f"
I20260809 22:36:49.521025 31022 raft_consensus.cc:3037] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:49.521874 31323 raft_consensus.cc:2454] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f2cef657584b66b306c3c17ab02d69 in term 1.
I20260809 22:36:49.522332 31094 leader_election.cc:302] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f2cef657584b66b306c3c17ab02d69, aedac66ab5bf4a2fa628c1bf1e894d71; no voters:
I20260809 22:36:49.522698 31402 raft_consensus.cc:2781] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:49.523209 31402 raft_consensus.cc:686] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 1 LEADER]: Becoming Leader. State: Replica: a2f2cef657584b66b306c3c17ab02d69, State: Running, Role: LEADER
I20260809 22:36:49.523880 31022 raft_consensus.cc:2454] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2f2cef657584b66b306c3c17ab02d69 in term 1.
I20260809 22:36:49.523787 31402 consensus_queue.cc:237] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.529217 30853 catalog_manager.cc:5495] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66). New cstate: current_term: 1 leader_uuid: "a2f2cef657584b66b306c3c17ab02d69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
W20260809 22:36:49.554168 31377 tablet.cc:2369] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:36:49.580212 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.574116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.28.243.66:43441)
0809 22:36:49.574402 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.574410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.574567 (+ 157us) server_negotiation.cc:407] Connection header received
0809 22:36:49.575023 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.575028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.575112 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.575269 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:49.575645 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.576389 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.577828 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.578263 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.578397 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.578922 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:49.579549 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:49.579719 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.579841 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.579931 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 22:36:49.590807 31323 raft_consensus.cc:1261] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Refusing update from remote peer a2f2cef657584b66b306c3c17ab02d69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:49.592083 31406 consensus_queue.cc:1035] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:49.592798 31022 raft_consensus.cc:1261] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Refusing update from remote peer a2f2cef657584b66b306c3c17ab02d69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:49.593709 31406 consensus_queue.cc:1035] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:49.609418 31409 mvcc.cc:205] Tried to move back new op lower bound from 7316746279269314560 to 7316746279014387712. Current Snapshot: MvccSnapshot[applied={T|T < 7316746279269314560}]
I20260809 22:36:49.641021 30853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:49.641425 30853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:49.645645 30853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36: Sending DeleteTablet for 3 replicas of tablet 382654e1ecd949b091265bd1a44a15c2
I20260809 22:36:49.647279 30996 tablet_service.cc:1514] Processing DeleteTablet for tablet 382654e1ecd949b091265bd1a44a15c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:49 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:47322
I20260809 22:36:49.647634 31147 tablet_service.cc:1514] Processing DeleteTablet for tablet 382654e1ecd949b091265bd1a44a15c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:49 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:60190
I20260809 22:36:49.648329 31300 tablet_service.cc:1514] Processing DeleteTablet for tablet 382654e1ecd949b091265bd1a44a15c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:49 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:46976
I20260809 22:36:49.649080 30853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:36:49.650216 30853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:49.660612 30995 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4b0f28809d4f8586597f179aa18ce3 (DEFAULT_TABLE table=test-table [id=71376e492c32495e822c2daaa746c221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:49.661366 30995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4b0f28809d4f8586597f179aa18ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.661424 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4b0f28809d4f8586597f179aa18ce3 (DEFAULT_TABLE table=test-table [id=71376e492c32495e822c2daaa746c221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:49.661972 31421 tablet_replica.cc:330] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: stopping tablet replica
I20260809 22:36:49.662259 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4b0f28809d4f8586597f179aa18ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.662823 31421 raft_consensus.cc:2229] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:36:49.663220 31422 tablet_replica.cc:330] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: stopping tablet replica
I20260809 22:36:49.663664 31422 raft_consensus.cc:2229] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:49.663968 31421 raft_consensus.cc:2258] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:49.665223 31146 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4b0f28809d4f8586597f179aa18ce3 (DEFAULT_TABLE table=test-table [id=71376e492c32495e822c2daaa746c221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:49.665387 31422 raft_consensus.cc:2258] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:49.665788 31146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4b0f28809d4f8586597f179aa18ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.666846 31421 ts_tablet_manager.cc:1905] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:49.669025 31422 ts_tablet_manager.cc:1905] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:49.672880 31399 tablet_bootstrap.cc:492] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: Bootstrap starting.
I20260809 22:36:49.675655 31399 tablet_bootstrap.cc:654] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.676841 31395 tablet_bootstrap.cc:492] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: Bootstrap starting.
I20260809 22:36:49.678718 31395 tablet_bootstrap.cc:654] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.680532 31396 tablet_bootstrap.cc:492] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: Bootstrap starting.
I20260809 22:36:49.682804 31396 tablet_bootstrap.cc:654] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.685102 31420 tablet_replica.cc:330] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: stopping tablet replica
I20260809 22:36:49.691829 31420 raft_consensus.cc:2229] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:49.692648 31420 raft_consensus.cc:2258] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:49.693743 31422 ts_tablet_manager.cc:1918] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:36:49.693971 31421 ts_tablet_manager.cc:1918] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:36:49.694192 31421 log.cc:1196] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/382654e1ecd949b091265bd1a44a15c2
I20260809 22:36:49.694221 31422 log.cc:1196] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/382654e1ecd949b091265bd1a44a15c2
I20260809 22:36:49.694890 31421 ts_tablet_manager.cc:1939] T 382654e1ecd949b091265bd1a44a15c2 P a2f2cef657584b66b306c3c17ab02d69: Deleting consensus metadata
I20260809 22:36:49.695044 31422 ts_tablet_manager.cc:1939] T 382654e1ecd949b091265bd1a44a15c2 P aedac66ab5bf4a2fa628c1bf1e894d71: Deleting consensus metadata
I20260809 22:36:49.695822 31420 ts_tablet_manager.cc:1905] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:49.697057 30840 catalog_manager.cc:4841] TS aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67:34293): tablet 382654e1ecd949b091265bd1a44a15c2 (table test-table [id=f91c4146fbeb4974bba1138a2b2d36d5]) successfully deleted
I20260809 22:36:49.697444 30840 catalog_manager.cc:4841] TS a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66:43441): tablet 382654e1ecd949b091265bd1a44a15c2 (table test-table [id=f91c4146fbeb4974bba1138a2b2d36d5]) successfully deleted
I20260809 22:36:49.697988 31396 tablet_bootstrap.cc:492] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: No bootstrap required, opened a new log
I20260809 22:36:49.698225 31396 ts_tablet_manager.cc:1397] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.009s
I20260809 22:36:49.699460 31396 raft_consensus.cc:348] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.699859 31396 raft_consensus.cc:374] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.700052 31396 raft_consensus.cc:729] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f2cef657584b66b306c3c17ab02d69, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.700475 31396 consensus_queue.cc:260] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.701349 31396 ts_tablet_manager.cc:1428] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 22:36:49.703967 31395 tablet_bootstrap.cc:492] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: No bootstrap required, opened a new log
I20260809 22:36:49.704156 31395 ts_tablet_manager.cc:1397] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.008s
I20260809 22:36:49.705173 31395 raft_consensus.cc:348] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.705428 31395 raft_consensus.cc:374] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.705543 31395 raft_consensus.cc:729] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56601b7d9544443aab8fa90dc3cbcc1f, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.705857 31395 consensus_queue.cc:260] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.707068 31420 ts_tablet_manager.cc:1918] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:36:49.707331 31420 log.cc:1196] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/382654e1ecd949b091265bd1a44a15c2
I20260809 22:36:49.707675 31395 ts_tablet_manager.cc:1428] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 22:36:49.708160 31420 ts_tablet_manager.cc:1939] T 382654e1ecd949b091265bd1a44a15c2 P 56601b7d9544443aab8fa90dc3cbcc1f: Deleting consensus metadata
I20260809 22:36:49.709028 31399 tablet_bootstrap.cc:492] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: No bootstrap required, opened a new log
I20260809 22:36:49.709213 31399 ts_tablet_manager.cc:1397] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.006s
I20260809 22:36:49.710122 31399 raft_consensus.cc:348] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.710381 31399 raft_consensus.cc:374] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.710494 31399 raft_consensus.cc:729] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedac66ab5bf4a2fa628c1bf1e894d71, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.710718 30841 catalog_manager.cc:4841] TS 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65:38287): tablet 382654e1ecd949b091265bd1a44a15c2 (table test-table [id=f91c4146fbeb4974bba1138a2b2d36d5]) successfully deleted
I20260809 22:36:49.710772 31399 consensus_queue.cc:260] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.711404 31399 ts_tablet_manager.cc:1428] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260809 22:36:49.721716 31401 raft_consensus.cc:482] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:49.722019 31401 raft_consensus.cc:504] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.723732 31401 leader_election.cc:288] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66:43441), aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67:34293)
I20260809 22:36:49.733851 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.725243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:49153 (local address 127.28.243.67:34293)
0809 22:36:49.725431 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.725438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.725626 (+ 188us) server_negotiation.cc:407] Connection header received
0809 22:36:49.725894 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.725900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.725989 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.726106 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:49.726806 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.727610 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.731108 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.732454 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.732672 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.732780 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.733344 (+ 564us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.733518 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 22:36:49.733994 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.724863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:34293 (local address 127.28.243.65:49153)
0809 22:36:49.725524 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.725544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.725707 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.726145 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.726150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.726182 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:49.726619 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.726629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.727755 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.727764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.730892 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.730910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.732820 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.732840 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.733065 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.733108 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.733283 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.733505 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":192,"thread_start_us":574,"threads_started":1}
I20260809 22:36:49.734444 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.724337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:44495 (local address 127.28.243.66:43441)
0809 22:36:49.724618 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.724625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.724844 (+ 219us) server_negotiation.cc:407] Connection header received
0809 22:36:49.725191 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.725200 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.725326 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.725410 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:49.726392 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.727676 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.731108 (+ 3432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.732661 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.732836 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.732898 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.733527 (+ 629us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.733672 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260809 22:36:49.734797 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4b0f28809d4f8586597f179aa18ce3" candidate_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" is_pre_election: true
I20260809 22:36:49.735113 31323 raft_consensus.cc:2454] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56601b7d9544443aab8fa90dc3cbcc1f in term 0.
I20260809 22:36:49.735432 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.724032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:43441 (local address 127.28.243.65:44495)
0809 22:36:49.724701 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.724727 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.724975 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.725654 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.725661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.725691 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:49.726129 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.726144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.727837 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.727842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.730892 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.730910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.733053 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.733058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.733313 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.733354 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.734748 (+ 1394us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.734982 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":282,"thread_start_us":302,"threads_started":1}
I20260809 22:36:49.735673 30946 leader_election.cc:302] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56601b7d9544443aab8fa90dc3cbcc1f, aedac66ab5bf4a2fa628c1bf1e894d71; no voters:
I20260809 22:36:49.736187 31401 raft_consensus.cc:2781] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:49.736335 31401 raft_consensus.cc:482] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:49.736438 31401 raft_consensus.cc:3037] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:49.740317 31401 raft_consensus.cc:504] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.741087 31169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4b0f28809d4f8586597f179aa18ce3" candidate_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f2cef657584b66b306c3c17ab02d69" is_pre_election: true
I20260809 22:36:49.741340 31401 leader_election.cc:288] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [CANDIDATE]: Term 1 election: Requested vote from peers a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66:43441), aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67:34293)
I20260809 22:36:49.741631 31169 raft_consensus.cc:2454] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56601b7d9544443aab8fa90dc3cbcc1f in term 0.
I20260809 22:36:49.742054 31323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4b0f28809d4f8586597f179aa18ce3" candidate_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71"
I20260809 22:36:49.742312 31323 raft_consensus.cc:3037] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:49.742534 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4b0f28809d4f8586597f179aa18ce3" candidate_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f2cef657584b66b306c3c17ab02d69"
I20260809 22:36:49.742844 31168 raft_consensus.cc:3037] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:49.745338 31168 raft_consensus.cc:2454] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56601b7d9544443aab8fa90dc3cbcc1f in term 1.
I20260809 22:36:49.745802 31323 raft_consensus.cc:2454] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56601b7d9544443aab8fa90dc3cbcc1f in term 1.
I20260809 22:36:49.745821 30946 leader_election.cc:302] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56601b7d9544443aab8fa90dc3cbcc1f, a2f2cef657584b66b306c3c17ab02d69; no voters:
I20260809 22:36:49.746223 31401 raft_consensus.cc:2781] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:49.753161 31401 raft_consensus.cc:686] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 LEADER]: Becoming Leader. State: Replica: 56601b7d9544443aab8fa90dc3cbcc1f, State: Running, Role: LEADER
I20260809 22:36:49.754024 31401 consensus_queue.cc:237] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.758430 30853 catalog_manager.cc:5495] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65). New cstate: current_term: 1 leader_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
Aug 09 22:36:49 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315007, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:49.812479 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.795347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45337 (local address 127.0.0.1:43050)
0809 22:36:49.796097 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:49.796121 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:49.796276 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:49.796704 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:49.796709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:49.797602 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:49.798254 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.798273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.799817 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.799821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.800950 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:49.800963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.802108 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.802113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:49.802243 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.803013 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:49.803033 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:49.805625 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:49.809626 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:49.809648 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:49.809655 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:49.810131 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:49.810573 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:49.810577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:49.810579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:49.810740 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:49.811406 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:49.811412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:49.811614 (+ 202us) client_negotiation.cc:769] Sending connection context
0809 22:36:49.811813 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 22:36:49.812102 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":367,"threads_started":1}
I20260809 22:36:49.812781 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:49.795280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.28.243.126:45337)
0809 22:36:49.795471 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:49.795476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:49.796228 (+ 752us) server_negotiation.cc:407] Connection header received
0809 22:36:49.796446 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:49.796452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:49.796533 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:49.796661 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:49.798446 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.799654 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.801110 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:49.801913 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:49.802063 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:49.805839 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:49.805846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:49.805852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:49.805891 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:49.809460 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.810259 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.810265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.810269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.810429 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:49.810849 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:49.810854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:49.810858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:49.811249 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:49.811407 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:49.811919 (+ 512us) server_negotiation.cc:299] Negotiation successful
0809 22:36:49.812318 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":63}
I20260809 22:36:49.819110 30853 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:43050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 22:36:49.820740 30853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:49.828759 30996 tablet_service.cc:1467] Processing CreateTablet for tablet 043cb56a68b14e6f83e1d5ebcad3b0d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b80fdd1371843449a25f7d4706e3bdb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:36:49.829089 31146 tablet_service.cc:1467] Processing CreateTablet for tablet 043cb56a68b14e6f83e1d5ebcad3b0d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b80fdd1371843449a25f7d4706e3bdb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:36:49.829020 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 043cb56a68b14e6f83e1d5ebcad3b0d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b80fdd1371843449a25f7d4706e3bdb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:36:49.829429 30996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043cb56a68b14e6f83e1d5ebcad3b0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.829622 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043cb56a68b14e6f83e1d5ebcad3b0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.829639 31146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 043cb56a68b14e6f83e1d5ebcad3b0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:49.837862 31396 tablet_bootstrap.cc:492] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: Bootstrap starting.
I20260809 22:36:49.837862 31395 tablet_bootstrap.cc:492] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: Bootstrap starting.
I20260809 22:36:49.840004 31395 tablet_bootstrap.cc:654] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.840025 31396 tablet_bootstrap.cc:654] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.842427 31399 tablet_bootstrap.cc:492] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: Bootstrap starting.
I20260809 22:36:49.842878 31396 tablet_bootstrap.cc:492] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: No bootstrap required, opened a new log
I20260809 22:36:49.843068 31396 ts_tablet_manager.cc:1397] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 22:36:49.843942 31396 raft_consensus.cc:348] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.844233 31396 raft_consensus.cc:374] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.844340 31396 raft_consensus.cc:729] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f2cef657584b66b306c3c17ab02d69, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.844429 31399 tablet_bootstrap.cc:654] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:49.844710 31396 consensus_queue.cc:260] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.845464 31396 ts_tablet_manager.cc:1428] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 22:36:49.845323 31402 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:49.845732 31402 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:49.846613 31395 tablet_bootstrap.cc:492] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: No bootstrap required, opened a new log
I20260809 22:36:49.846802 31395 ts_tablet_manager.cc:1397] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260809 22:36:49.847546 31395 raft_consensus.cc:348] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.847827 31395 raft_consensus.cc:374] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.847918 31395 raft_consensus.cc:729] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56601b7d9544443aab8fa90dc3cbcc1f, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.848217 31395 consensus_queue.cc:260] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.848903 31401 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:49.849110 31395 ts_tablet_manager.cc:1428] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 22:36:49.849269 31401 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:49.857069 31399 tablet_bootstrap.cc:492] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: No bootstrap required, opened a new log
I20260809 22:36:49.857324 31399 ts_tablet_manager.cc:1397] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260809 22:36:49.858039 31399 raft_consensus.cc:348] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.858305 31399 raft_consensus.cc:374] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:49.858392 31399 raft_consensus.cc:729] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedac66ab5bf4a2fa628c1bf1e894d71, State: Initialized, Role: FOLLOWER
I20260809 22:36:49.858659 31399 consensus_queue.cc:260] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:49.859330 31399 ts_tablet_manager.cc:1428] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 22:36:49.859202 31403 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:49.859573 31403 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: This TxnStatusTablet replica's current role is: FOLLOWER
W20260809 22:36:49.923695 31071 tablet.cc:2369] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:36:50.141013 31403 raft_consensus.cc:482] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:50.141286 31403 raft_consensus.cc:504] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:50.143253 31403 leader_election.cc:288] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65:38287), a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66:43441)
I20260809 22:36:50.161680 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.144150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:37683 (local address 127.28.243.66:43441)
0809 22:36:50.144408 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.144417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.155634 (+ 11217us) server_negotiation.cc:407] Connection header received
0809 22:36:50.155762 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.155769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.155988 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.156130 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:50.156721 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.157649 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.159540 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.160387 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.160625 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.160712 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.161189 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.161342 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 22:36:50.162108 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.143656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:43441 (local address 127.28.243.67:37683)
0809 22:36:50.155486 (+ 11830us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:50.155507 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:50.155625 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:50.156145 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:50.156151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:50.156185 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:50.156505 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.156516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.157804 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.157810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.159359 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.159372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.160651 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.160659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.160939 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.160986 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 22:36:50.161119 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 22:36:50.161280 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11676,"thread_start_us":10534,"threads_started":1}
I20260809 22:36:50.162757 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.143717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:34575 (local address 127.28.243.65:38287)
0809 22:36:50.154946 (+ 11229us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.154955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.154982 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:36:50.155071 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.155078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.155171 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.155304 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:50.155906 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.157168 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.160349 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.161587 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.161770 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.161851 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.162311 (+ 460us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.162464 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11064,"thread_start_us":11172,"threads_started":1}
I20260809 22:36:50.162810 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.143388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38287 (local address 127.28.243.67:34575)
0809 22:36:50.154561 (+ 11173us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:50.154596 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:50.154781 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:50.155399 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:50.155405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:50.155432 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:50.155739 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.155752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.157358 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.157366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.160102 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.160138 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.162004 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.162009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.162150 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.162173 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 22:36:50.162283 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 22:36:50.162416 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10983,"mutex_wait_us":33,"thread_start_us":11774,"threads_started":1}
I20260809 22:36:50.163278 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043cb56a68b14e6f83e1d5ebcad3b0d6" candidate_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f2cef657584b66b306c3c17ab02d69" is_pre_election: true
I20260809 22:36:50.163422 31022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043cb56a68b14e6f83e1d5ebcad3b0d6" candidate_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" is_pre_election: true
I20260809 22:36:50.163686 31022 raft_consensus.cc:2454] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedac66ab5bf4a2fa628c1bf1e894d71 in term 0.
I20260809 22:36:50.163692 31168 raft_consensus.cc:2454] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedac66ab5bf4a2fa628c1bf1e894d71 in term 0.
I20260809 22:36:50.164283 31250 leader_election.cc:302] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 56601b7d9544443aab8fa90dc3cbcc1f, a2f2cef657584b66b306c3c17ab02d69, aedac66ab5bf4a2fa628c1bf1e894d71; no voters:
I20260809 22:36:50.164851 31403 raft_consensus.cc:2781] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:50.165089 31403 raft_consensus.cc:482] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:50.165218 31403 raft_consensus.cc:3037] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:50.167976 31403 raft_consensus.cc:504] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:50.168733 31403 leader_election.cc:288] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [CANDIDATE]: Term 1 election: Requested vote from peers 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65:38287), a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66:43441)
I20260809 22:36:50.169322 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043cb56a68b14e6f83e1d5ebcad3b0d6" candidate_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2f2cef657584b66b306c3c17ab02d69"
I20260809 22:36:50.169612 31168 raft_consensus.cc:3037] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:50.169668 31022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "043cb56a68b14e6f83e1d5ebcad3b0d6" candidate_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56601b7d9544443aab8fa90dc3cbcc1f"
I20260809 22:36:50.169921 31022 raft_consensus.cc:3037] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:50.173625 31168 raft_consensus.cc:2454] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedac66ab5bf4a2fa628c1bf1e894d71 in term 1.
I20260809 22:36:50.173821 31022 raft_consensus.cc:2454] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedac66ab5bf4a2fa628c1bf1e894d71 in term 1.
I20260809 22:36:50.174181 31249 leader_election.cc:302] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2f2cef657584b66b306c3c17ab02d69, aedac66ab5bf4a2fa628c1bf1e894d71; no voters:
I20260809 22:36:50.174896 31403 raft_consensus.cc:2781] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:50.176926 31403 raft_consensus.cc:686] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 LEADER]: Becoming Leader. State: Replica: aedac66ab5bf4a2fa628c1bf1e894d71, State: Running, Role: LEADER
I20260809 22:36:50.177402 31403 consensus_queue.cc:237] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } }
I20260809 22:36:50.181938 31443 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: TxnStatusTablet state changed. Reason: New leader aedac66ab5bf4a2fa628c1bf1e894d71. Latest consensus state: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:50.182282 31443 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:36:50.184960 30854 catalog_manager.cc:5495] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 reported cstate change: term changed from 0 to 1, leader changed from <none> to aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67). New cstate: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260809 22:36:50.193226 31445 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:36:50.193572 31445 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 22:36:50.239112 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.231449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:34293 (local address 127.0.0.1:46982)
0809 22:36:50.231689 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:50.231717 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:50.231840 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:50.232447 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:50.232451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:50.232485 (+ 34us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:36:50.232782 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.232797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.234504 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.234511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.235910 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.235927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.237171 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.237177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.237310 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.237382 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:36:50.238570 (+ 1188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:36:50.238593 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 22:36:50.238661 (+ 68us) client_negotiation.cc:240] Negotiation successful
0809 22:36:50.238796 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 22:36:50.239133 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.231620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.28.243.67:34293)
0809 22:36:50.231811 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.231817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.231840 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:36:50.232102 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.232109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.232246 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.232407 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:50.232950 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.234328 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.236067 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.236900 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.237035 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.237592 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:50.238435 (+ 843us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:50.238588 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.238704 (+ 116us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.238802 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:36:50.245187 31022 raft_consensus.cc:1261] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Refusing update from remote peer aedac66ab5bf4a2fa628c1bf1e894d71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:50.245544 31168 raft_consensus.cc:1261] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Refusing update from remote peer aedac66ab5bf4a2fa628c1bf1e894d71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:50.245965 31403 consensus_queue.cc:1035] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:50.246516 31443 consensus_queue.cc:1035] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:50.254583 31402 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: TxnStatusTablet state changed. Reason: New leader aedac66ab5bf4a2fa628c1bf1e894d71. Latest consensus state: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:50.255034 31402 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:50.257728 31448 mvcc.cc:205] Tried to move back new op lower bound from 7316746281956081664 to 7316746281691598848. Current Snapshot: MvccSnapshot[applied={T|T < 7316746281956081664}]
I20260809 22:36:50.258550 31403 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:50.258893 31403 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:36:50.264945 31401 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: TxnStatusTablet state changed. Reason: New leader aedac66ab5bf4a2fa628c1bf1e894d71. Latest consensus state: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:50.265223 31401 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:50.269722 31443 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:50.270221 31443 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P aedac66ab5bf4a2fa628c1bf1e894d71: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:36:50.274143 31402 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:50.274750 31402 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P a2f2cef657584b66b306c3c17ab02d69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:50.279220 31401 tablet_replica.cc:439] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56601b7d9544443aab8fa90dc3cbcc1f" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38287 } } peers { permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 } } peers { permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 } } }
I20260809 22:36:50.279560 31401 tablet_replica.cc:442] T 043cb56a68b14e6f83e1d5ebcad3b0d6 P 56601b7d9544443aab8fa90dc3cbcc1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:50.287470 31401 consensus_queue.cc:1035] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedac66ab5bf4a2fa628c1bf1e894d71" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:50.288292 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.279484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.28.243.65:38287)
0809 22:36:50.279845 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.279850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.280723 (+ 873us) server_negotiation.cc:407] Connection header received
0809 22:36:50.280990 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.280995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.281122 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.281280 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:50.281938 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.283161 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.285102 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.285846 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.286005 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.286605 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:50.287390 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:50.287561 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.287691 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.287785 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260809 22:36:50.297578 31401 consensus_queue.cc:1035] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2f2cef657584b66b306c3c17ab02d69" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:50.333699 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.325977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:43287 (local address 127.28.243.67:34293)
0809 22:36:50.326165 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.326171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.326464 (+ 293us) server_negotiation.cc:407] Connection header received
0809 22:36:50.326673 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.326678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.326760 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.326853 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:50.327454 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.328688 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.331040 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.331895 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.332070 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.332726 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:50.332851 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:50.332945 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.333151 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.333302 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 22:36:50.333799 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.325670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:34293 (local address 127.28.243.65:43287)
0809 22:36:50.326341 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:50.326370 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:50.326536 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:50.326902 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:50.326908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:50.326923 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:36:50.327268 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.327291 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.328849 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.328856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.330867 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.330886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.332110 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.332117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.332341 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.332451 (+ 110us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:36:50.332948 (+ 497us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:36:50.332963 (+ 15us) client_negotiation.cc:769] Sending connection context
0809 22:36:50.333128 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 22:36:50.333292 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":486,"threads_started":1}
I20260809 22:36:50.388069 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.379744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:54499 (local address 127.28.243.65:38287)
0809 22:36:50.379939 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.379945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.380886 (+ 941us) server_negotiation.cc:407] Connection header received
0809 22:36:50.381178 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.381184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.381274 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.381390 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:50.382371 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.383571 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.385845 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.386519 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.386637 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.387276 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:50.387396 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:50.387474 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.387768 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.387843 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 22:36:50.388355 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.379828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38287 (local address 127.28.243.67:54499)
0809 22:36:50.380740 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:50.380769 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:50.380983 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:50.381639 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:50.381644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:50.381657 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:36:50.382044 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.382057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.383777 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.383784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.385664 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:50.385682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.386806 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.386812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:50.387020 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.387097 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:36:50.387500 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:36:50.387522 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 22:36:50.387712 (+ 190us) client_negotiation.cc:240] Negotiation successful
0809 22:36:50.387892 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":445,"threads_started":1}
I20260809 22:36:50.434266 30854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:50.434587 30854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:50.438723 30854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2d8d3071c9c42bfb73c02c9d7a53b36: Sending DeleteTablet for 3 replicas of tablet 6a4b0f28809d4f8586597f179aa18ce3
I20260809 22:36:50.439656 30996 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4b0f28809d4f8586597f179aa18ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:50 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:47322
I20260809 22:36:50.440196 31146 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4b0f28809d4f8586597f179aa18ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:50 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:60190
I20260809 22:36:50.440918 31474 tablet_replica.cc:330] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: stopping tablet replica
I20260809 22:36:50.440982 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4b0f28809d4f8586597f179aa18ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:50 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:46976
I20260809 22:36:50.441124 31474 raft_consensus.cc:2229] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:50.441383 31474 raft_consensus.cc:2258] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:50.445114 31475 tablet_replica.cc:330] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: stopping tablet replica
I20260809 22:36:50.445400 31475 raft_consensus.cc:2229] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:50.445751 31475 raft_consensus.cc:2258] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:50.449350 31474 ts_tablet_manager.cc:1905] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:50.451982 31473 tablet_replica.cc:330] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: stopping tablet replica
I20260809 22:36:50.452371 31473 raft_consensus.cc:2229] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:36:50.453320 31473 raft_consensus.cc:2258] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:50.454789 31473 ts_tablet_manager.cc:1905] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:50.457727 31475 ts_tablet_manager.cc:1905] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:50.463275 31473 ts_tablet_manager.cc:1918] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:50.463544 31473 log.cc:1196] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/6a4b0f28809d4f8586597f179aa18ce3
I20260809 22:36:50.464202 31473 ts_tablet_manager.cc:1939] T 6a4b0f28809d4f8586597f179aa18ce3 P 56601b7d9544443aab8fa90dc3cbcc1f: Deleting consensus metadata
I20260809 22:36:50.465628 30841 catalog_manager.cc:4841] TS 56601b7d9544443aab8fa90dc3cbcc1f (127.28.243.65:38287): tablet 6a4b0f28809d4f8586597f179aa18ce3 (table test-table [id=71376e492c32495e822c2daaa746c221]) successfully deleted
I20260809 22:36:50.465754 31475 ts_tablet_manager.cc:1918] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:50.465960 31475 log.cc:1196] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/6a4b0f28809d4f8586597f179aa18ce3
I20260809 22:36:50.466415 31474 ts_tablet_manager.cc:1918] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:50.466498 31475 ts_tablet_manager.cc:1939] T 6a4b0f28809d4f8586597f179aa18ce3 P aedac66ab5bf4a2fa628c1bf1e894d71: Deleting consensus metadata
I20260809 22:36:50.466671 31474 log.cc:1196] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/6a4b0f28809d4f8586597f179aa18ce3
I20260809 22:36:50.467259 31474 ts_tablet_manager.cc:1939] T 6a4b0f28809d4f8586597f179aa18ce3 P a2f2cef657584b66b306c3c17ab02d69: Deleting consensus metadata
I20260809 22:36:50.467912 30840 catalog_manager.cc:4841] TS aedac66ab5bf4a2fa628c1bf1e894d71 (127.28.243.67:34293): tablet 6a4b0f28809d4f8586597f179aa18ce3 (table test-table [id=71376e492c32495e822c2daaa746c221]) successfully deleted
I20260809 22:36:50.468760 30840 catalog_manager.cc:4841] TS a2f2cef657584b66b306c3c17ab02d69 (127.28.243.66:43441): tablet 6a4b0f28809d4f8586597f179aa18ce3 (table test-table [id=71376e492c32495e822c2daaa746c221]) successfully deleted
Aug 09 22:36:50 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:36:50.498417 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.463186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47352 (local address 127.28.243.65:38287)
0809 22:36:50.463388 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.463395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.474035 (+ 10640us) server_negotiation.cc:407] Connection header received
0809 22:36:50.474309 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.474315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.474415 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.474527 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:50.477385 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.478695 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.480842 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.481678 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.481851 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.487821 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:50.487839 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:50.487862 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:50.487925 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:50.493449 (+ 5524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.494593 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.494605 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.494608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.494800 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.495299 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.495304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.495307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.495931 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:50.496286 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.497803 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.498057 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260809 22:36:50.499281 31065 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47352
I20260809 22:36:50.543789 31480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.518586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.28.243.126:45337)
0809 22:36:50.525731 (+ 7145us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.525737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.525759 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:36:50.525821 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.525826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.525916 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.526047 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:50.528242 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.529659 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.531503 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.532281 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.532446 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.535753 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:50.535761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:50.535767 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:50.535819 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:50.540134 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.541174 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.541181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.541186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.541455 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.541964 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.541970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.541974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.542396 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:50.542733 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.543050 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.543328 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7006,"thread_start_us":6852,"threads_started":1}
W20260809 22:36:50.544447 30854 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43054
I20260809 22:36:50.547676 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30927
I20260809 22:36:50.569459 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.550560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.28.243.65:38287)
0809 22:36:50.550786 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.550792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.550814 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:36:50.550885 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.550891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.551063 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.551191 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:50.553041 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.554563 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.556811 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.557581 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.557697 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.561392 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:50.561410 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:50.561419 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:50.561471 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:50.565744 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.566736 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.566743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.566748 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.566952 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.567483 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.567489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.567494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.567953 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:50.568242 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.568726 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.568997 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 22:36:50.570204 31065 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47362
W20260809 22:36:50.570859 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30927: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:36:50.571077 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 30927
I20260809 22:36:50.593516 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31074
Aug 09 22:36:50 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:36:50.613600 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.594419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.28.243.66:43441)
0809 22:36:50.594662 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.594669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.594692 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:36:50.594831 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.594837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.594935 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.595068 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:50.596395 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.597612 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.599170 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.599868 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.600005 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.605022 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:50.605058 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:50.605079 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:50.605133 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:50.609668 (+ 4535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.610563 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.610575 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.610579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.610790 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.611537 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.611543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.611547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.612114 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:50.612506 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.612885 (+ 379us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.613170 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 22:36:50.614578 31219 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60200
W20260809 22:36:50.615439 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:36:50.615651 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 31074
I20260809 22:36:50.641863 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31229
Aug 09 22:36:50 dist-test-slave-rgb1 krb5kdc[30805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:36:50.661731 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:50.642781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.28.243.67:34293)
0809 22:36:50.643023 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:50.643028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:50.643052 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:36:50.643261 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:50.643266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:50.643397 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:50.643507 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:50.645349 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.646183 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.648083 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:50.648805 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:50.648965 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:50.654904 (+ 5939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:50.654921 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:50.654947 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:50.654990 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:50.658325 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.659299 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.659307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.659309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.659492 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:50.659955 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:50.659960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:50.659963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:50.660430 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:50.660953 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:50.661090 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 22:36:50.661296 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260809 22:36:50.662592 31370 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46992
W20260809 22:36:50.663650 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31229: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:36:50.663857 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 31229
I20260809 22:36:50.687532 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 30818
I20260809 22:36:50.688795 30896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:50.812273 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 30818
2026-08-09T22:36:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5235 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:36:50 dist-test-slave-rgb1 krb5kdc[31485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:36:50 dist-test-slave-rgb1 krb5kdc[31485](info): set up 2 sockets
Aug 09 22:36:50 dist-test-slave-rgb1 krb5kdc[31485](info): commencing operation
krb5kdc: starting...
W20260809 22:36:51.995893 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:36:52 dist-test-slave-rgb1 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:36:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:36:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:36:52.241626 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41925
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:41925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:52.583772 31500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:52.584128 31500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:52.584288 31500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:52.591806 31500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:36:52.591982 31500 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 22:36:52.592077 31500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:52.592191 31500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:52.592279 31500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:36:52.592374 31500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:36:52.602883 31500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:41925
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:52.604179 31500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:52.605547 31500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:52.617749 31506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:52.617763 31509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:52.617951 31507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:52.618680 31500 server_base.cc:1034] running on GCE node
I20260809 22:36:52.619498 31500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:52.620998 31500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:52.622262 31500 hybrid_clock.cc:648] HybridClock initialized: now 1786315012622196 us; error 40 us; skew 500 ppm
I20260809 22:36:52.622634 31500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:52 dist-test-slave-rgb1 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315012, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:52.626382 31500 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:36:52.628132 31500 webserver.cc:460] Webserver started at http://127.28.243.126:44579/ using document root <none> and password file <none>
I20260809 22:36:52.628964 31500 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:52.629099 31500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:52.629441 31500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:52.632686 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "a9ce188d68ed46b988934064a0259f6a"
format_stamp: "Formatted at 2026-08-09 22:36:52 on dist-test-slave-rgb1"
server_key: "8a9c675825f866dbdaad1f75976c1cc4"
server_key_iv: "a20b1d66739707d669283e17d82d5092"
server_key_version: "encryptionkey@0"
I20260809 22:36:52.633525 31500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "a9ce188d68ed46b988934064a0259f6a"
format_stamp: "Formatted at 2026-08-09 22:36:52 on dist-test-slave-rgb1"
server_key: "8a9c675825f866dbdaad1f75976c1cc4"
server_key_iv: "a20b1d66739707d669283e17d82d5092"
server_key_version: "encryptionkey@0"
I20260809 22:36:52.639127 31500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 22:36:52.643046 31516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:52.644490 31500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:36:52.644722 31500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "a9ce188d68ed46b988934064a0259f6a"
format_stamp: "Formatted at 2026-08-09 22:36:52 on dist-test-slave-rgb1"
server_key: "8a9c675825f866dbdaad1f75976c1cc4"
server_key_iv: "a20b1d66739707d669283e17d82d5092"
server_key_version: "encryptionkey@0"
I20260809 22:36:52.644961 31500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:52.667915 31500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:52.672545 31500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:52.672957 31500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:52.695621 31500 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:41925
I20260809 22:36:52.695644 31584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:41925 every 8 connection(s)
I20260809 22:36:52.697094 31500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:36:52.701263 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:52.705090 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 31500
I20260809 22:36:52.705318 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:36:52.706059 29645 external_mini_cluster.cc:1428] Setting key a0b64d720fd24cf1f087355fbd4636ee
I20260809 22:36:52.714254 31585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Bootstrap starting.
I20260809 22:36:52.718197 31585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:52.719579 31585 log.cc:824] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:52.722880 31585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: No bootstrap required, opened a new log
Aug 09 22:36:52 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:52.733733 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:52.708275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.28.243.126:41925)
0809 22:36:52.709221 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:52.709243 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:52.709296 (+ 53us) server_negotiation.cc:407] Connection header received
0809 22:36:52.712292 (+ 2996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:52.712326 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:52.712930 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:52.713431 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:52.715049 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:52.717175 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:52.718686 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:52.719818 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:52.720014 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:52.724275 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:52.724286 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:52.724305 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:52.724357 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:52.728676 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:52.729442 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:52.729456 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:52.729465 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:52.729702 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:52.730166 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:52.730170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:52.730174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:52.730948 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:52.731227 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:52.731688 (+ 461us) server_negotiation.cc:299] Negotiation successful
0809 22:36:52.732190 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":417,"threads_started":1}
I20260809 22:36:52.734211 31585 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ce188d68ed46b988934064a0259f6a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41925 } }
I20260809 22:36:52.734603 31585 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:52.734735 31585 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ce188d68ed46b988934064a0259f6a, State: Initialized, Role: FOLLOWER
I20260809 22:36:52.735443 31585 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ce188d68ed46b988934064a0259f6a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41925 } }
I20260809 22:36:52.735774 31585 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:36:52.735930 31585 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:36:52.736137 31585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:52.738719 31585 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ce188d68ed46b988934064a0259f6a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41925 } }
I20260809 22:36:52.739375 31585 leader_election.cc:302] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9ce188d68ed46b988934064a0259f6a; no voters:
I20260809 22:36:52.740132 31585 leader_election.cc:288] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:36:52.740260 31590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:52.741040 31590 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [term 1 LEADER]: Becoming Leader. State: Replica: a9ce188d68ed46b988934064a0259f6a, State: Running, Role: LEADER
I20260809 22:36:52.741683 31590 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ce188d68ed46b988934064a0259f6a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41925 } }
I20260809 22:36:52.742317 31585 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:36:52.744904 31592 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9ce188d68ed46b988934064a0259f6a. Latest consensus state: current_term: 1 leader_uuid: "a9ce188d68ed46b988934064a0259f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ce188d68ed46b988934064a0259f6a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41925 } } }
I20260809 22:36:52.745386 31592 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:52.745934 31591 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9ce188d68ed46b988934064a0259f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9ce188d68ed46b988934064a0259f6a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41925 } } }
I20260809 22:36:52.746255 31591 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a [sys.catalog]: This master's current role is: LEADER
I20260809 22:36:52.746451 31597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:36:52.751794 31597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:36:52.761227 31597 catalog_manager.cc:1347] Generated new cluster ID: 6f212ab239ae4a58b37c63dfc0457d1b
I20260809 22:36:52.761354 31597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:36:52.787878 31597 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:36:52.789722 31597 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:36:52.796829 31597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Generated new TSK 0
I20260809 22:36:52.797914 31597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:36:52.854511 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41925
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:53.209489 31613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:53.209828 31613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:53.209961 31613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:53.218690 31613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:53.218822 31613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:53.218917 31613 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 22:36:53.218973 31613 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:36:53.219038 31613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:36:53.229285 31613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:41925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:53.230623 31613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:53.231985 31613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:53.246968 31618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:53.246939 31621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:53.247352 31619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:53.247663 31613 server_base.cc:1034] running on GCE node
I20260809 22:36:53.248620 31613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:53.249763 31613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:53.250975 31613 hybrid_clock.cc:648] HybridClock initialized: now 1786315013250901 us; error 70 us; skew 500 ppm
I20260809 22:36:53.251379 31613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:53 dist-test-slave-rgb1 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:53.254978 31613 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:36:53.256644 31613 webserver.cc:460] Webserver started at http://127.28.243.65:34021/ using document root <none> and password file <none>
I20260809 22:36:53.257377 31613 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:53.257488 31613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:53.257771 31613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:53.261240 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "fa6fbd29677f44f99f7f61154e1d38c7"
format_stamp: "Formatted at 2026-08-09 22:36:53 on dist-test-slave-rgb1"
server_key: "80ab0b054c509088f89802daae27ab00"
server_key_iv: "d77065c2148979c8d01e6e6c96afa2e5"
server_key_version: "encryptionkey@0"
I20260809 22:36:53.262133 31613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "fa6fbd29677f44f99f7f61154e1d38c7"
format_stamp: "Formatted at 2026-08-09 22:36:53 on dist-test-slave-rgb1"
server_key: "80ab0b054c509088f89802daae27ab00"
server_key_iv: "d77065c2148979c8d01e6e6c96afa2e5"
server_key_version: "encryptionkey@0"
I20260809 22:36:53.267977 31613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 22:36:53.272279 31628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:53.274228 31613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260809 22:36:53.274442 31613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "fa6fbd29677f44f99f7f61154e1d38c7"
format_stamp: "Formatted at 2026-08-09 22:36:53 on dist-test-slave-rgb1"
server_key: "80ab0b054c509088f89802daae27ab00"
server_key_iv: "d77065c2148979c8d01e6e6c96afa2e5"
server_key_version: "encryptionkey@0"
I20260809 22:36:53.274701 31613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:53.300326 31613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:53.305598 31613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:53.305918 31613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:53.308845 31613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:53.308979 31613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:53.309129 31613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:53.309270 31613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 09 22:36:53 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:53.335882 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.309444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:51389 (local address 127.28.243.126:41925)
0809 22:36:53.310230 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:53.310237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:53.311259 (+ 1022us) server_negotiation.cc:407] Connection header received
0809 22:36:53.313606 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:53.313612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:53.313753 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:53.313894 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:53.317101 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.318105 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.321696 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.322434 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.322623 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.328931 (+ 6308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:53.328938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:53.328941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:53.328983 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:53.331978 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.332913 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.332919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.332923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.333114 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.333854 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.333862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.333867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.334223 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:53.334393 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:53.335230 (+ 837us) server_negotiation.cc:299] Negotiation successful
0809 22:36:53.335480 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":526,"threads_started":1}
I20260809 22:36:53.337255 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.310147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41925 (local address 127.28.243.65:51389)
0809 22:36:53.311061 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:53.311118 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:53.313272 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:53.314282 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:53.314300 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:53.315414 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:53.316815 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.316848 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.318309 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.318314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.321395 (+ 3081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.321418 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.323075 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.323087 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.323485 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.324503 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:53.324587 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:53.328417 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:53.332154 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.332164 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.332183 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.332764 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.333291 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.333298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.333308 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.333697 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.334367 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:53.334379 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:53.334824 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 22:36:53.335115 (+ 291us) client_negotiation.cc:240] Negotiation successful
0809 22:36:53.335679 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":400,"threads_started":1}
I20260809 22:36:53.362454 31613 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:35685
I20260809 22:36:53.362486 31758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:35685 every 8 connection(s)
I20260809 22:36:53.364326 31613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:36:53.364789 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.346636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:53141 (local address 127.28.243.126:41925)
0809 22:36:53.346869 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:53.346877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:53.347692 (+ 815us) server_negotiation.cc:407] Connection header received
0809 22:36:53.347964 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:53.347970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:53.348080 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:53.348214 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:53.350212 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.350931 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.353520 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.353962 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.354069 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.357467 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:53.357478 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:53.357484 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:53.357537 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:53.361538 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.362642 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.362648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.362651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.362811 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.363272 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.363276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.363278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.363596 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:53.363723 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:53.364225 (+ 502us) server_negotiation.cc:299] Negotiation successful
0809 22:36:53.364446 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:36:53.365095 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.346789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41925 (local address 127.28.243.65:53141)
0809 22:36:53.347565 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:53.347586 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:53.347833 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:53.348258 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:53.348266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:53.349403 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:53.350022 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.350041 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.351096 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.351105 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.353365 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.353384 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.354188 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.354199 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.354441 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.355352 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:53.355385 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:53.357145 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:53.361748 (+ 4603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.361757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.361763 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.362465 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.362968 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.362972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.362976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.363133 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.363726 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:53.363734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:53.363950 (+ 216us) client_negotiation.cc:769] Sending connection context
0809 22:36:53.364107 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 22:36:53.364537 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":509,"threads_started":1}
I20260809 22:36:53.366856 31759 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41925
I20260809 22:36:53.367151 31759 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:53.367898 31759 heartbeater.cc:511] Master 127.28.243.126:41925 requested a full tablet report, sending...
I20260809 22:36:53.369837 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 31613
I20260809 22:36:53.370113 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:36:53.370463 31533 ts_manager.cc:194] Registered new tserver with Master: fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685)
I20260809 22:36:53.370814 29645 external_mini_cluster.cc:1428] Setting key aa81212f667abaa2d2b228f0840d812a
I20260809 22:36:53.372349 31533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:51389
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:36:53.425416 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41925
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:53.770852 31764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:53.771169 31764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:53.771366 31764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:53.779016 31764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:53.779140 31764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:53.779309 31764 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 22:36:53.779387 31764 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:36:53.779511 31764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:36:53.790606 31764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:41925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:53.792026 31764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:53.793440 31764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:53.807919 31770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:53.808043 31769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:53.808043 31772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:53.810734 31764 server_base.cc:1034] running on GCE node
I20260809 22:36:53.811586 31764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:53.812727 31764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:53.813988 31764 hybrid_clock.cc:648] HybridClock initialized: now 1786315013813913 us; error 70 us; skew 500 ppm
I20260809 22:36:53.814435 31764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:53 dist-test-slave-rgb1 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:53.818235 31764 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:36:53.820166 31764 webserver.cc:460] Webserver started at http://127.28.243.66:36135/ using document root <none> and password file <none>
I20260809 22:36:53.821018 31764 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:53.821193 31764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:53.821559 31764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:53.825042 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "4ec428191bb749d78083ffb619ca913b"
format_stamp: "Formatted at 2026-08-09 22:36:53 on dist-test-slave-rgb1"
server_key: "9405086ded8352a2d11ccefef163ad0d"
server_key_iv: "59dade1d8c5e469529fa23a3e9de4200"
server_key_version: "encryptionkey@0"
I20260809 22:36:53.825953 31764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "4ec428191bb749d78083ffb619ca913b"
format_stamp: "Formatted at 2026-08-09 22:36:53 on dist-test-slave-rgb1"
server_key: "9405086ded8352a2d11ccefef163ad0d"
server_key_iv: "59dade1d8c5e469529fa23a3e9de4200"
server_key_version: "encryptionkey@0"
I20260809 22:36:53.832074 31764 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 22:36:53.836292 31779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:53.838102 31764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 22:36:53.838297 31764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "4ec428191bb749d78083ffb619ca913b"
format_stamp: "Formatted at 2026-08-09 22:36:53 on dist-test-slave-rgb1"
server_key: "9405086ded8352a2d11ccefef163ad0d"
server_key_iv: "59dade1d8c5e469529fa23a3e9de4200"
server_key_version: "encryptionkey@0"
I20260809 22:36:53.838528 31764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:53.853137 31764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:53.857988 31764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:53.858312 31764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:53.861657 31764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:53.861828 31764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:53.862032 31764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:53.862157 31764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:36:53 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:53.891552 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.862878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41925 (local address 127.28.243.66:53959)
0809 22:36:53.863871 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:53.863934 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:53.866351 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:53.867737 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:53.867753 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:53.868887 (+ 1134us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:53.869963 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.869993 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.871796 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.871804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.874052 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.874074 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.875353 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.875362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.875713 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.876722 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:53.876766 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:53.880850 (+ 4084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:53.885371 (+ 4521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.885381 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.885403 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.886186 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.886748 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.886753 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.886757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.887117 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.888001 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:53.888009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:53.888666 (+ 657us) client_negotiation.cc:769] Sending connection context
0809 22:36:53.889130 (+ 464us) client_negotiation.cc:240] Negotiation successful
0809 22:36:53.889746 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":474,"threads_started":1}
I20260809 22:36:53.892714 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.862084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:53959 (local address 127.28.243.126:41925)
0809 22:36:53.862344 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:53.862357 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:53.864111 (+ 1754us) server_negotiation.cc:407] Connection header received
0809 22:36:53.866788 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:53.866797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:53.866945 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:53.867061 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:53.870209 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.871586 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.874288 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.875066 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.875229 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.881202 (+ 5973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:53.881214 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:53.881221 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:53.881280 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:53.885199 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.886376 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.886382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.886386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.886580 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.887305 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.887310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.887314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.887712 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:53.887898 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:53.892026 (+ 4128us) server_negotiation.cc:299] Negotiation successful
0809 22:36:53.892336 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 22:36:53.905619 31764 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:33401
I20260809 22:36:53.905707 31907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:33401 every 8 connection(s)
I20260809 22:36:53.907241 31764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:36:53.909725 31908 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41925
I20260809 22:36:53.909838 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 31764
I20260809 22:36:53.909976 31908 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:53.910041 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:36:53.910486 29645 external_mini_cluster.cc:1428] Setting key be2f2247c7a97888fb36e4d4db498727
I20260809 22:36:53.911146 31908 heartbeater.cc:511] Master 127.28.243.126:41925 requested a full tablet report, sending...
I20260809 22:36:53.913089 31533 ts_manager.cc:194] Registered new tserver with Master: 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401)
I20260809 22:36:53.914323 31533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:53959
I20260809 22:36:53.916548 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.899327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41925 (local address 127.28.243.66:60011)
0809 22:36:53.900211 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:53.900240 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:53.900485 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:53.901003 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:53.901010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:53.901837 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:53.902700 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.902725 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.904025 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.904033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.906038 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:53.906054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.906812 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.906820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:53.907011 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.907729 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:53.907757 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:53.909194 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:53.913317 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.913323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.913326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.913853 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.914357 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:53.914364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:53.914369 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:53.914537 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:53.915351 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:53.915356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:53.915561 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 22:36:53.915713 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 22:36:53.916064 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":495,"threads_started":1}
I20260809 22:36:53.917008 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:53.899093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:60011 (local address 127.28.243.126:41925)
0809 22:36:53.899320 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:53.899326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:53.900343 (+ 1017us) server_negotiation.cc:407] Connection header received
0809 22:36:53.900721 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:53.900727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:53.900802 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:53.900922 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:53.902920 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.903874 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.906186 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:53.906604 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:53.906711 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:53.909371 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:53.909375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:53.909378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:53.909411 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:53.913137 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.914018 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.914024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.914027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.914204 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:53.914670 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:53.914675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:53.914678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:53.915138 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:53.915286 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:53.915941 (+ 655us) server_negotiation.cc:299] Negotiation successful
0809 22:36:53.916254 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:36:53.966200 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41925
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:36:54.322072 31912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:36:54.322386 31912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:36:54.322491 31912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:36:54.330580 31912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:36:54.330729 31912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:36:54.330832 31912 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 22:36:54.330902 31912 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:36:54.330981 31912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:36:54.342067 31912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:41925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:36:54.343497 31912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:36:54.345016 31912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:36:54.359380 31917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:54.359476 31918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:36:54.359983 31920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:36:54.360492 31912 server_base.cc:1034] running on GCE node
I20260809 22:36:54.361419 31912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:36:54.362988 31912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:36:54.364394 31912 hybrid_clock.cc:648] HybridClock initialized: now 1786315014364266 us; error 126 us; skew 500 ppm
I20260809 22:36:54.364804 31912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:36:54 dist-test-slave-rgb1 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:36:54.369323 31912 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:36:54.374236 31912 webserver.cc:460] Webserver started at http://127.28.243.67:37881/ using document root <none> and password file <none>
I20260809 22:36:54.375504 31912 fs_manager.cc:359] Metadata directory not provided
I20260809 22:36:54.375708 31912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:36:54.375952 31759 heartbeater.cc:503] Master 127.28.243.126:41925 was elected leader, sending a full tablet report...
I20260809 22:36:54.376312 31912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:36:54.381714 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
format_stamp: "Formatted at 2026-08-09 22:36:54 on dist-test-slave-rgb1"
server_key: "8e6f4b067bfc51750075d21694297a5f"
server_key_iv: "497120bda9c03fc2da4e466356630a3c"
server_key_version: "encryptionkey@0"
I20260809 22:36:54.382887 31912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
format_stamp: "Formatted at 2026-08-09 22:36:54 on dist-test-slave-rgb1"
server_key: "8e6f4b067bfc51750075d21694297a5f"
server_key_iv: "497120bda9c03fc2da4e466356630a3c"
server_key_version: "encryptionkey@0"
I20260809 22:36:54.391194 31912 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260809 22:36:54.396799 31927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:54.398805 31912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 22:36:54.399036 31912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
format_stamp: "Formatted at 2026-08-09 22:36:54 on dist-test-slave-rgb1"
server_key: "8e6f4b067bfc51750075d21694297a5f"
server_key_iv: "497120bda9c03fc2da4e466356630a3c"
server_key_version: "encryptionkey@0"
I20260809 22:36:54.399340 31912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:36:54.426555 31912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:36:54.431829 31912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:36:54.432206 31912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:36:54.435473 31912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:36:54.435659 31912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:36:54.435822 31912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:36:54.435933 31912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:36:54 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:54.463037 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.436033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:51195 (local address 127.28.243.126:41925)
0809 22:36:54.437124 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.437134 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.438399 (+ 1265us) server_negotiation.cc:407] Connection header received
0809 22:36:54.442270 (+ 3871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.442276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.442384 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.442518 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:54.445168 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.446165 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.448144 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.449016 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.449163 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.453946 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:54.453955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:54.453963 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:54.454028 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:54.457756 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.458844 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.458855 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.458862 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.459149 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.459708 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.459718 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.459725 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.460360 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:54.460642 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.461927 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.462295 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":747,"threads_started":1}
I20260809 22:36:54.464150 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.436827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41925 (local address 127.28.243.67:51195)
0809 22:36:54.438145 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.438225 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.441747 (+ 3522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.443040 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.443050 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.443978 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:54.444820 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.444864 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.446356 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.446361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.447950 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.447966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.449289 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.449298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.449558 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.450635 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:54.450659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:54.453611 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:54.457938 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:54.457949 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:54.457969 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:54.458607 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:54.459300 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:54.459305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:54.459309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:54.459540 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:54.460703 (+ 1163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:54.460710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:54.461124 (+ 414us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.461397 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.462466 (+ 1069us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":608,"threads_started":1}
I20260809 22:36:54.496308 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.472147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41925 (local address 127.28.243.67:57147)
0809 22:36:54.477369 (+ 5222us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.477402 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.477629 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.478039 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.478043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.478752 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:54.479114 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.479125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.480995 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.481005 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.483641 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.483670 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.484654 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.484666 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.485062 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.485915 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:54.485947 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:54.487425 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:54.492025 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:54.492034 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:54.492039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:54.492947 (+ 908us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:54.493497 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:54.493508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:54.493512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:54.493740 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:54.494610 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:54.494615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:54.494973 (+ 358us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.495247 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.495711 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4973,"thread_start_us":4747,"threads_started":1}
I20260809 22:36:54.497638 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.472200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:57147 (local address 127.28.243.126:41925)
0809 22:36:54.472492 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.472502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.477572 (+ 5070us) server_negotiation.cc:407] Connection header received
0809 22:36:54.477794 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.477804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.477881 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.478005 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:54.479283 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.480801 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.483880 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.484352 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.484475 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.487701 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:54.487708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:54.487711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:54.487748 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:54.491833 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.493154 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.493161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.493165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.493348 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.493945 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.493950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.493953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.494457 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:54.494647 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.496826 (+ 2179us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.497179 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:36:54.508333 31912 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:36919
I20260809 22:36:54.508401 32056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:36919 every 8 connection(s)
I20260809 22:36:54.509931 31912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:36:54.512210 32057 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41925
I20260809 22:36:54.512461 32057 heartbeater.cc:464] Registering TS with master...
I20260809 22:36:54.513136 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 31912
I20260809 22:36:54.513193 32057 heartbeater.cc:511] Master 127.28.243.126:41925 requested a full tablet report, sending...
I20260809 22:36:54.513327 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:36:54.513825 29645 external_mini_cluster.cc:1428] Setting key a445612c51d67b5f2a5ff83cbe035075
I20260809 22:36:54.514737 31533 ts_manager.cc:194] Registered new tserver with Master: f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:54.515818 31533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:51195
I20260809 22:36:54.517457 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 22:36:54 dist-test-slave-rgb1 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 22:36:54 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:54.576871 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.560146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.28.243.126:41925)
0809 22:36:54.560472 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.560477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.560499 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:36:54.560726 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.560731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.560821 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.560918 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:54.562592 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.563489 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.565357 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.565866 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.566046 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.570806 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:54.570812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:54.570816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:54.570864 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:54.573947 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.574747 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.574752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.574755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.574917 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.575376 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.575380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.575382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.575703 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:54.575958 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.576299 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.576495 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260809 22:36:54.581694 31534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:36:54.589180 31534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:54.626564 32074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.602099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:33401 (local address 127.0.0.1:49004)
0809 22:36:54.613484 (+ 11385us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.613571 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.613873 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.617306 (+ 3433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.617319 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.617362 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:54.618499 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.618655 (+ 156us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.620924 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.620932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.623667 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.623685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.625546 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.625553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.625745 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.625798 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.625947 (+ 149us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.626158 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10974,"thread_start_us":16216,"threads_started":1}
I20260809 22:36:54.627666 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.602331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.28.243.66:33401)
0809 22:36:54.616093 (+ 13762us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.616106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.616151 (+ 45us) server_negotiation.cc:407] Connection header received
0809 22:36:54.616302 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.616315 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.616681 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.616969 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:54.619111 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.620747 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.623864 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.625327 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.625490 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.627143 (+ 1653us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.627212 (+ 69us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.627379 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13534,"thread_start_us":15576,"threads_started":1}
I20260809 22:36:54.628381 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.602275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:35685 (local address 127.0.0.1:42034)
0809 22:36:54.620069 (+ 17794us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.620097 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.620237 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.620853 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.620858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.620883 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:54.621363 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.621377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.623149 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.623154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.625146 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.625161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.627491 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.627498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.627660 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.627705 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.627806 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.627977 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17674,"mutex_wait_us":6,"thread_start_us":10593,"threads_started":1}
I20260809 22:36:54.628971 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.602793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.28.243.65:35685)
0809 22:36:54.619977 (+ 17184us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.619986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.620216 (+ 230us) server_negotiation.cc:407] Connection header received
0809 22:36:54.620395 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.620406 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.620674 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.620875 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:54.621543 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.622960 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.625320 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.627237 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.627419 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.627639 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.628467 (+ 828us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.628689 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17008,"thread_start_us":16846,"threads_started":1}
I20260809 22:36:54.629070 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.614245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36919 (local address 127.0.0.1:59802)
0809 22:36:54.620312 (+ 6067us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.620345 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.620469 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.621418 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.621423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.621454 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:54.621858 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.621869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.623912 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.623915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.625462 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.625479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.627931 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.627939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.628078 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.628104 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.628208 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.628346 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5888,"thread_start_us":3738,"threads_started":1}
I20260809 22:36:54.630630 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.614693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.28.243.67:36919)
0809 22:36:54.619512 (+ 4819us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.619523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.620460 (+ 937us) server_negotiation.cc:407] Connection header received
0809 22:36:54.620643 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.620650 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.620883 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.621089 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:54.622105 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.623702 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.625675 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.627402 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.627531 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.627753 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.628266 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.628446 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4620,"thread_start_us":5164,"threads_started":1}
I20260809 22:36:54.631975 31832 tablet_service.cc:1467] Processing CreateTablet for tablet 63403f5adb654ba580cfb59c16190df3 (DEFAULT_TABLE table=test-table [id=9bedf10fb3f74a11ae6906b2a55185e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:54.632732 31683 tablet_service.cc:1467] Processing CreateTablet for tablet 63403f5adb654ba580cfb59c16190df3 (DEFAULT_TABLE table=test-table [id=9bedf10fb3f74a11ae6906b2a55185e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:54.633281 31832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63403f5adb654ba580cfb59c16190df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:54.634119 31683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63403f5adb654ba580cfb59c16190df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:54.634251 31986 tablet_service.cc:1467] Processing CreateTablet for tablet 63403f5adb654ba580cfb59c16190df3 (DEFAULT_TABLE table=test-table [id=9bedf10fb3f74a11ae6906b2a55185e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:54.635910 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63403f5adb654ba580cfb59c16190df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:54.647060 32080 tablet_bootstrap.cc:492] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Bootstrap starting.
I20260809 22:36:54.651392 32080 tablet_bootstrap.cc:654] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:54.652942 32080 log.cc:824] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:54.656096 32083 tablet_bootstrap.cc:492] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Bootstrap starting.
I20260809 22:36:54.656646 32080 tablet_bootstrap.cc:492] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: No bootstrap required, opened a new log
I20260809 22:36:54.656924 32081 tablet_bootstrap.cc:492] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Bootstrap starting.
I20260809 22:36:54.656966 32080 ts_tablet_manager.cc:1397] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260809 22:36:54.660420 32083 tablet_bootstrap.cc:654] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:54.661087 32081 tablet_bootstrap.cc:654] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:54.661897 32083 log.cc:824] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:54.662370 32081 log.cc:824] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Log is configured to *not* fsync() on all Append() calls
I20260809 22:36:54.666024 32083 tablet_bootstrap.cc:492] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: No bootstrap required, opened a new log
I20260809 22:36:54.666353 32081 tablet_bootstrap.cc:492] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: No bootstrap required, opened a new log
I20260809 22:36:54.666432 32083 ts_tablet_manager.cc:1397] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260809 22:36:54.666663 32081 ts_tablet_manager.cc:1397] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260809 22:36:54.669617 32080 raft_consensus.cc:348] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.670109 32080 raft_consensus.cc:374] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:54.670207 32080 raft_consensus.cc:729] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Initialized, Role: FOLLOWER
I20260809 22:36:54.670984 32080 consensus_queue.cc:260] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.674723 32083 raft_consensus.cc:348] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.675107 32083 raft_consensus.cc:374] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:54.675235 32083 raft_consensus.cc:729] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bbf9e6fd8c40299a50aafd34e0aa31, State: Initialized, Role: FOLLOWER
I20260809 22:36:54.675941 31908 heartbeater.cc:503] Master 127.28.243.126:41925 was elected leader, sending a full tablet report...
I20260809 22:36:54.675985 32083 consensus_queue.cc:260] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.676594 32080 ts_tablet_manager.cc:1428] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Time spent starting tablet: real 0.019s user 0.011s sys 0.009s
I20260809 22:36:54.677531 32057 heartbeater.cc:503] Master 127.28.243.126:41925 was elected leader, sending a full tablet report...
I20260809 22:36:54.678118 32083 ts_tablet_manager.cc:1428] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260809 22:36:54.679507 32081 raft_consensus.cc:348] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.679930 32081 raft_consensus.cc:374] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:54.680044 32081 raft_consensus.cc:729] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6fbd29677f44f99f7f61154e1d38c7, State: Initialized, Role: FOLLOWER
I20260809 22:36:54.680824 32081 consensus_queue.cc:260] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.682705 32081 ts_tablet_manager.cc:1428] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260809 22:36:54.761946 32058 tablet.cc:2369] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:36:54.832778 32086 raft_consensus.cc:482] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:54.833361 32086 raft_consensus.cc:504] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.835426 32086 leader_election.cc:288] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:54.845654 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.837401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:34585 (local address 127.28.243.67:36919)
0809 22:36:54.837637 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.837646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.838248 (+ 602us) server_negotiation.cc:407] Connection header received
0809 22:36:54.838448 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.838456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.838553 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.838773 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:54.839569 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.840648 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.843564 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.844683 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.844837 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.844928 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.845328 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.845444 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 22:36:54.846056 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.837243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36919 (local address 127.28.243.66:34585)
0809 22:36:54.838106 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.838125 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.838280 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.838809 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.838815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.838839 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:54.839370 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.839385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.840840 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.840847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.843364 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.843382 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.844918 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.844924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.845138 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.845177 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.845280 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.845424 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":47,"thread_start_us":412,"threads_started":1}
I20260809 22:36:54.847146 32089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.836092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:35685 (local address 127.28.243.66:50797)
0809 22:36:54.837004 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.837024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.837249 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.837816 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.837824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.837864 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:54.838338 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.838353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.840331 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.840337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.843365 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.843382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.846356 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.846363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.846558 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.846592 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.846705 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.846823 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":79,"thread_start_us":614,"threads_started":1}
I20260809 22:36:54.847352 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.836134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:50797 (local address 127.28.243.65:35685)
0809 22:36:54.836420 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.836428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.837136 (+ 708us) server_negotiation.cc:407] Connection header received
0809 22:36:54.837519 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.837527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.837653 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.837771 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:54.838548 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.840015 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.844697 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.845969 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.846178 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.846259 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.846883 (+ 624us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.847036 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 22:36:54.847395 32010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63403f5adb654ba580cfb59c16190df3" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" is_pre_election: true
I20260809 22:36:54.848070 32010 raft_consensus.cc:2454] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 0.
I20260809 22:36:54.849205 31783 leader_election.cc:302] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, f6bbf9e6fd8c40299a50aafd34e0aa31; no voters:
I20260809 22:36:54.849664 32086 raft_consensus.cc:2781] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:54.849853 32086 raft_consensus.cc:482] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:54.849987 32086 raft_consensus.cc:3037] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:54.850660 31712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63403f5adb654ba580cfb59c16190df3" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" is_pre_election: true
I20260809 22:36:54.851276 31712 raft_consensus.cc:2454] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 0.
I20260809 22:36:54.853122 32086 raft_consensus.cc:504] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.853796 32086 leader_election.cc:288] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 election: Requested vote from peers fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:54.855011 31712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63403f5adb654ba580cfb59c16190df3" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6fbd29677f44f99f7f61154e1d38c7"
I20260809 22:36:54.855224 31712 raft_consensus.cc:3037] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:54.855075 32010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63403f5adb654ba580cfb59c16190df3" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
I20260809 22:36:54.855646 32010 raft_consensus.cc:3037] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:54.857847 31712 raft_consensus.cc:2454] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 1.
I20260809 22:36:54.858373 31783 leader_election.cc:302] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:54.858484 32010 raft_consensus.cc:2454] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 1.
I20260809 22:36:54.858865 32086 raft_consensus.cc:2781] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:54.859486 32086 raft_consensus.cc:686] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 1 LEADER]: Becoming Leader. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Running, Role: LEADER
I20260809 22:36:54.860095 32086 consensus_queue.cc:237] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.865537 31534 catalog_manager.cc:5495] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ec428191bb749d78083ffb619ca913b (127.28.243.66). New cstate: current_term: 1 leader_uuid: "4ec428191bb749d78083ffb619ca913b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } health_report { overall_health: UNKNOWN } } }
W20260809 22:36:54.867877 31760 tablet.cc:2369] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:36:54.909979 31909 tablet.cc:2369] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 22:36:54 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315012, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:54.967839 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.951774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41925 (local address 127.0.0.1:37094)
0809 22:36:54.952536 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:54.952611 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:54.952754 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:54.953188 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:54.953192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:54.953903 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:36:54.954257 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.954274 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.955343 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.955347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.956441 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:54.956453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.957494 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.957500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:54.957644 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.958389 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:36:54.958409 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:36:54.961217 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:36:54.964701 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:54.964716 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:54.964724 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:54.965310 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:54.965779 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:36:54.965783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:36:54.965785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:36:54.966033 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:36:54.966720 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:36:54.966728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:36:54.966897 (+ 169us) client_negotiation.cc:769] Sending connection context
0809 22:36:54.967071 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 22:36:54.967377 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":438,"threads_started":1}
I20260809 22:36:54.967935 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:54.951756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.28.243.126:41925)
0809 22:36:54.952005 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:54.952010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:54.952717 (+ 707us) server_negotiation.cc:407] Connection header received
0809 22:36:54.952864 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:54.952868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:54.953007 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:54.953102 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:54.954433 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.955197 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.956674 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:54.957276 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:54.957404 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:54.961485 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:54.961491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:54.961494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:54.961532 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:54.964441 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.965476 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.965481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.965484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.965638 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:54.966197 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:54.966200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:54.966202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:54.966519 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:54.966658 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:54.967206 (+ 548us) server_negotiation.cc:299] Negotiation successful
0809 22:36:54.967585 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":68}
I20260809 22:36:54.973775 31534 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:37094:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 22:36:54.975030 31534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:54.983351 31683 tablet_service.cc:1467] Processing CreateTablet for tablet 56cacfd16ef541dd905a0a33dff64c31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6bd82e2ed91e480fb3ec861ba2b7feb1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:36:54.983703 31832 tablet_service.cc:1467] Processing CreateTablet for tablet 56cacfd16ef541dd905a0a33dff64c31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6bd82e2ed91e480fb3ec861ba2b7feb1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:36:54.983803 31986 tablet_service.cc:1467] Processing CreateTablet for tablet 56cacfd16ef541dd905a0a33dff64c31 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6bd82e2ed91e480fb3ec861ba2b7feb1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:36:54.984630 31832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cacfd16ef541dd905a0a33dff64c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:54.984651 31683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cacfd16ef541dd905a0a33dff64c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:54.984756 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cacfd16ef541dd905a0a33dff64c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:54.990693 32083 tablet_bootstrap.cc:492] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: Bootstrap starting.
I20260809 22:36:54.992275 32083 tablet_bootstrap.cc:654] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:54.992910 32080 tablet_bootstrap.cc:492] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: Bootstrap starting.
I20260809 22:36:54.993516 32081 tablet_bootstrap.cc:492] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: Bootstrap starting.
I20260809 22:36:54.994961 32080 tablet_bootstrap.cc:654] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:54.995602 32081 tablet_bootstrap.cc:654] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:54.996098 32083 tablet_bootstrap.cc:492] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: No bootstrap required, opened a new log
I20260809 22:36:54.996264 32083 ts_tablet_manager.cc:1397] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260809 22:36:54.997527 32080 tablet_bootstrap.cc:492] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: No bootstrap required, opened a new log
I20260809 22:36:54.997366 32083 raft_consensus.cc:348] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.997689 32083 raft_consensus.cc:374] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:54.997707 32080 ts_tablet_manager.cc:1397] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 22:36:54.997812 32083 raft_consensus.cc:729] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bbf9e6fd8c40299a50aafd34e0aa31, State: Initialized, Role: FOLLOWER
I20260809 22:36:54.998113 32083 consensus_queue.cc:260] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:54.998970 32083 ts_tablet_manager.cc:1428] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260809 22:36:54.998768 32087 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:54.999163 32087 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:54.999791 32080 raft_consensus.cc:348] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.000077 32080 raft_consensus.cc:374] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:55.000192 32080 raft_consensus.cc:729] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Initialized, Role: FOLLOWER
I20260809 22:36:55.000486 32080 consensus_queue.cc:260] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.000751 32081 tablet_bootstrap.cc:492] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: No bootstrap required, opened a new log
I20260809 22:36:55.000926 32081 ts_tablet_manager.cc:1397] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 22:36:55.000974 32086 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.001286 32086 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:55.001770 32080 ts_tablet_manager.cc:1428] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260809 22:36:55.001989 32081 raft_consensus.cc:348] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.003120 32081 raft_consensus.cc:374] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:55.003222 32081 raft_consensus.cc:729] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6fbd29677f44f99f7f61154e1d38c7, State: Initialized, Role: FOLLOWER
I20260809 22:36:55.003530 32081 consensus_queue.cc:260] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.004201 32081 ts_tablet_manager.cc:1428] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260809 22:36:55.004065 32088 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.004505 32088 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:55.138309 32088 raft_consensus.cc:482] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:55.138564 32088 raft_consensus.cc:504] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.140465 32088 leader_election.cc:288] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:55.148779 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.141036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:58243 (local address 127.28.243.67:36919)
0809 22:36:55.141269 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:55.141279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:55.142083 (+ 804us) server_negotiation.cc:407] Connection header received
0809 22:36:55.142492 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:55.142509 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:55.142616 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:55.142732 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:55.143353 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.144232 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.146760 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.147654 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.147834 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.147922 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:55.148236 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 22:36:55.148374 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 22:36:55.148953 32103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.140876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36919 (local address 127.28.243.65:58243)
0809 22:36:55.141910 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:55.141937 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:55.142262 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:55.142751 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:55.142756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:55.142782 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:55.143180 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.143201 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.144389 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.144397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.146588 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.146602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.147883 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.147890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.148058 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.148083 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 22:36:55.148196 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 22:36:55.148351 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":64,"thread_start_us":581,"threads_started":1}
I20260809 22:36:55.149441 32102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.140876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:33401 (local address 127.28.243.65:58875)
0809 22:36:55.142268 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:55.142284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:55.142439 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:55.143253 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:55.143258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:55.143284 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:36:55.143698 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.143711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.145068 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.145073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.146852 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.146866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.148310 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.148317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.148687 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.148718 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 22:36:55.148829 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 22:36:55.148985 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":113,"thread_start_us":963,"threads_started":1}
I20260809 22:36:55.149634 32010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cacfd16ef541dd905a0a33dff64c31" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" is_pre_election: true
I20260809 22:36:55.149744 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.141008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:58875 (local address 127.28.243.66:33401)
0809 22:36:55.142945 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:55.142949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:55.142966 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:36:55.143050 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:55.143053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:55.143108 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:55.143230 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:36:55.143897 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.144892 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.147068 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.148033 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.148269 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.148346 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:55.148862 (+ 516us) server_negotiation.cc:299] Negotiation successful
0809 22:36:55.149385 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1803,"thread_start_us":1642,"threads_started":1}
I20260809 22:36:55.149930 32010 raft_consensus.cc:2454] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 0.
I20260809 22:36:55.150430 31633 leader_election.cc:302] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6bbf9e6fd8c40299a50aafd34e0aa31, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:55.150663 31856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cacfd16ef541dd905a0a33dff64c31" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec428191bb749d78083ffb619ca913b" is_pre_election: true
I20260809 22:36:55.150873 32088 raft_consensus.cc:2781] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:55.151000 32088 raft_consensus.cc:482] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:55.151094 32088 raft_consensus.cc:3037] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:55.151137 31856 raft_consensus.cc:2454] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 0.
I20260809 22:36:55.153481 32088 raft_consensus.cc:504] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.154098 32088 leader_election.cc:288] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 election: Requested vote from peers 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:55.154661 32010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cacfd16ef541dd905a0a33dff64c31" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
I20260809 22:36:55.154744 31856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cacfd16ef541dd905a0a33dff64c31" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec428191bb749d78083ffb619ca913b"
I20260809 22:36:55.154929 32010 raft_consensus.cc:3037] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:55.154949 31856 raft_consensus.cc:3037] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:55.158121 32010 raft_consensus.cc:2454] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 1.
I20260809 22:36:55.158133 31856 raft_consensus.cc:2454] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 1.
I20260809 22:36:55.158658 31629 leader_election.cc:302] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:55.159093 32088 raft_consensus.cc:2781] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:55.159623 32088 raft_consensus.cc:686] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 LEADER]: Becoming Leader. State: Replica: fa6fbd29677f44f99f7f61154e1d38c7, State: Running, Role: LEADER
I20260809 22:36:55.160140 32088 consensus_queue.cc:237] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.163681 32105 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: TxnStatusTablet state changed. Reason: New leader fa6fbd29677f44f99f7f61154e1d38c7. Latest consensus state: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.164263 32105 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:36:55.164994 32107 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:36:55.165287 32107 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 22:36:55.165230 31532 catalog_manager.cc:5495] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } health_report { overall_health: UNKNOWN } } }
I20260809 22:36:55.199293 32098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.192114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:35685 (local address 127.0.0.1:42036)
0809 22:36:55.192400 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:55.192420 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:55.192661 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:55.193303 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:55.193307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:55.193319 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:36:55.193543 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.193553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.195148 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.195152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.196386 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.196400 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.197463 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.197468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.197629 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.197692 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:36:55.198791 (+ 1099us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:36:55.198815 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 22:36:55.198900 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 22:36:55.199047 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 22:36:55.199374 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.192266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.28.243.65:35685)
0809 22:36:55.192488 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:55.192496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:55.192523 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:36:55.192906 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:55.192915 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:55.193040 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:55.193211 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:55.193691 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.194994 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.196582 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.197245 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.197409 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.197909 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:55.198649 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:55.198806 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:55.198945 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 22:36:55.199038 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:36:55.208766 31856 raft_consensus.cc:1261] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Refusing update from remote peer fa6fbd29677f44f99f7f61154e1d38c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:55.208794 32010 raft_consensus.cc:1261] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Refusing update from remote peer fa6fbd29677f44f99f7f61154e1d38c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:55.209846 32088 consensus_queue.cc:1035] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:55.210175 32105 consensus_queue.cc:1035] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:55.218349 32087 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: TxnStatusTablet state changed. Reason: New leader fa6fbd29677f44f99f7f61154e1d38c7. Latest consensus state: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.218665 32087 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:55.221750 32110 mvcc.cc:205] Tried to move back new op lower bound from 7316746302284152832 to 7316746302100451328. Current Snapshot: MvccSnapshot[applied={T|T < 7316746302284152832}]
I20260809 22:36:55.222807 32105 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.223124 32105 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:36:55.231571 32087 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.231863 32087 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P f6bbf9e6fd8c40299a50aafd34e0aa31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:55.234382 32086 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: TxnStatusTablet state changed. Reason: New leader fa6fbd29677f44f99f7f61154e1d38c7. Latest consensus state: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.234835 32086 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:55.237775 32088 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.238210 32088 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P fa6fbd29677f44f99f7f61154e1d38c7: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:36:55.241030 32112 mvcc.cc:205] Tried to move back new op lower bound from 7316746302284152832 to 7316746302100451328. Current Snapshot: MvccSnapshot[applied={T|T < 7316746302284152832}]
I20260809 22:36:55.250085 32091 consensus_queue.cc:1035] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:55.251699 32086 tablet_replica.cc:439] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } } }
I20260809 22:36:55.252007 32086 tablet_replica.cc:442] T 56cacfd16ef541dd905a0a33dff64c31 P 4ec428191bb749d78083ffb619ca913b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:36:55.257325 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.237734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.28.243.66:33401)
0809 22:36:55.238522 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:55.238528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:55.249818 (+ 11290us) server_negotiation.cc:407] Connection header received
0809 22:36:55.250160 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:55.250168 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:55.250298 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:55.250517 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:55.251166 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.252377 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.254449 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.255139 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.255241 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.255955 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:55.256705 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:55.256860 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:55.256959 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 22:36:55.257046 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656}
I20260809 22:36:55.260713 32086 consensus_queue.cc:1035] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:55.285315 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.278432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:53771 (local address 127.28.243.65:35685)
0809 22:36:55.278647 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:55.278656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:55.279240 (+ 584us) server_negotiation.cc:407] Connection header received
0809 22:36:55.279447 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:55.279450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:55.279515 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:55.279592 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:55.280283 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.281223 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.283323 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.283817 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.283932 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.284529 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:55.284715 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:55.284805 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:55.284973 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 22:36:55.285050 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 22:36:55.285682 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.278418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:35685 (local address 127.28.243.66:53771)
0809 22:36:55.279154 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:55.279174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:55.279316 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:55.279705 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:55.279711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:55.279724 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:36:55.280116 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.280128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.281357 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.281363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.283169 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.283186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.284006 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.284013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.284189 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.284273 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:36:55.284823 (+ 550us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:36:55.284835 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 22:36:55.284938 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 22:36:55.285100 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":404,"threads_started":1}
I20260809 22:36:55.331000 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.323228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:52799 (local address 127.28.243.66:33401)
0809 22:36:55.323443 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:55.323449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:55.323910 (+ 461us) server_negotiation.cc:407] Connection header received
0809 22:36:55.324205 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:55.324211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:55.324324 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:55.324434 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:55.325304 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.326461 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.328832 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.329506 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.329645 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.330248 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:55.330374 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:55.330477 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:55.330596 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 22:36:55.330679 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 22:36:55.331310 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:55.322933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:33401 (local address 127.28.243.65:52799)
0809 22:36:55.323748 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:55.323777 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:55.324044 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:55.324499 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:55.324505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:55.324521 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:36:55.325078 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.325093 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.326679 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.326686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.328673 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:55.328690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:55.329833 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:55.329845 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:55.329987 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:55.330049 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:36:55.330484 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:36:55.330498 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 22:36:55.330576 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 22:36:55.330720 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":370,"threads_started":1}
I20260809 22:36:55.369691 31533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:55.370062 31533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:55.374675 31533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Sending DeleteTablet for 3 replicas of tablet 63403f5adb654ba580cfb59c16190df3
I20260809 22:36:55.376447 31832 tablet_service.cc:1514] Processing DeleteTablet for tablet 63403f5adb654ba580cfb59c16190df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:55 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:49004
I20260809 22:36:55.377820 31986 tablet_service.cc:1514] Processing DeleteTablet for tablet 63403f5adb654ba580cfb59c16190df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:55 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:59802
I20260809 22:36:55.387008 32133 tablet_replica.cc:330] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: stopping tablet replica
I20260809 22:36:55.387089 31533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 22:36:55.387605 32133 raft_consensus.cc:2229] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:36:55.388046 31683 tablet_service.cc:1514] Processing DeleteTablet for tablet 63403f5adb654ba580cfb59c16190df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:55 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:42034
I20260809 22:36:55.388284 32133 raft_consensus.cc:2258] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 22:36:55.389277 31533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:55.390486 32133 ts_tablet_manager.cc:1905] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:55.401135 32135 tablet_replica.cc:330] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: stopping tablet replica
I20260809 22:36:55.401633 32135 raft_consensus.cc:2229] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:55.401975 32135 raft_consensus.cc:2258] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:55.403930 32135 ts_tablet_manager.cc:1905] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:55.404317 32134 tablet_replica.cc:330] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: stopping tablet replica
I20260809 22:36:55.404835 32134 raft_consensus.cc:2229] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:55.405308 32134 raft_consensus.cc:2258] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:55.410349 32134 ts_tablet_manager.cc:1905] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:55.411743 32135 ts_tablet_manager.cc:1918] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:55.411942 32135 log.cc:1196] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/63403f5adb654ba580cfb59c16190df3
I20260809 22:36:55.412694 31683 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6e8be28c36400085db4ed408d8eee5 (DEFAULT_TABLE table=test-table [id=c8fa89b7146b4b77985f88629cba6cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:55.413535 31832 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6e8be28c36400085db4ed408d8eee5 (DEFAULT_TABLE table=test-table [id=c8fa89b7146b4b77985f88629cba6cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:55.413995 31832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6e8be28c36400085db4ed408d8eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:55.417263 31986 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6e8be28c36400085db4ed408d8eee5 (DEFAULT_TABLE table=test-table [id=c8fa89b7146b4b77985f88629cba6cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:55.417729 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6e8be28c36400085db4ed408d8eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:55.422791 32133 ts_tablet_manager.cc:1918] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:55.423038 32133 log.cc:1196] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/63403f5adb654ba580cfb59c16190df3
I20260809 22:36:55.424651 31683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6e8be28c36400085db4ed408d8eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:55.430764 32133 ts_tablet_manager.cc:1939] T 63403f5adb654ba580cfb59c16190df3 P 4ec428191bb749d78083ffb619ca913b: Deleting consensus metadata
I20260809 22:36:55.431905 32081 tablet_bootstrap.cc:492] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: Bootstrap starting.
I20260809 22:36:55.431948 32135 ts_tablet_manager.cc:1939] T 63403f5adb654ba580cfb59c16190df3 P fa6fbd29677f44f99f7f61154e1d38c7: Deleting consensus metadata
I20260809 22:36:55.433177 31517 catalog_manager.cc:4841] TS 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401): tablet 63403f5adb654ba580cfb59c16190df3 (table test-table [id=9bedf10fb3f74a11ae6906b2a55185e1]) successfully deleted
I20260809 22:36:55.433732 31520 catalog_manager.cc:4841] TS fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685): tablet 63403f5adb654ba580cfb59c16190df3 (table test-table [id=9bedf10fb3f74a11ae6906b2a55185e1]) successfully deleted
I20260809 22:36:55.433918 32081 tablet_bootstrap.cc:654] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:55.436467 32081 tablet_bootstrap.cc:492] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: No bootstrap required, opened a new log
I20260809 22:36:55.436691 32081 ts_tablet_manager.cc:1397] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260809 22:36:55.437541 32080 tablet_bootstrap.cc:492] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: Bootstrap starting.
I20260809 22:36:55.437522 32081 raft_consensus.cc:348] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.437788 32081 raft_consensus.cc:374] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:55.437877 32081 raft_consensus.cc:729] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6fbd29677f44f99f7f61154e1d38c7, State: Initialized, Role: FOLLOWER
I20260809 22:36:55.438138 32081 consensus_queue.cc:260] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.439432 32080 tablet_bootstrap.cc:654] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:55.439634 32081 ts_tablet_manager.cc:1428] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 22:36:55.445853 32083 tablet_bootstrap.cc:492] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: Bootstrap starting.
I20260809 22:36:55.447685 32083 tablet_bootstrap.cc:654] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:55.449713 32080 tablet_bootstrap.cc:492] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: No bootstrap required, opened a new log
I20260809 22:36:55.449936 32080 ts_tablet_manager.cc:1397] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20260809 22:36:55.450821 32080 raft_consensus.cc:348] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.451118 32080 raft_consensus.cc:374] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:55.451215 32080 raft_consensus.cc:729] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Initialized, Role: FOLLOWER
I20260809 22:36:55.452008 32080 consensus_queue.cc:260] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.453389 32080 ts_tablet_manager.cc:1428] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260809 22:36:55.457758 32083 tablet_bootstrap.cc:492] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: No bootstrap required, opened a new log
I20260809 22:36:55.457968 32083 ts_tablet_manager.cc:1397] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20260809 22:36:55.458762 32083 raft_consensus.cc:348] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.459033 32083 raft_consensus.cc:374] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:55.459122 32083 raft_consensus.cc:729] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bbf9e6fd8c40299a50aafd34e0aa31, State: Initialized, Role: FOLLOWER
I20260809 22:36:55.459582 32083 consensus_queue.cc:260] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.461565 32083 ts_tablet_manager.cc:1428] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 22:36:55.465704 32134 ts_tablet_manager.cc:1918] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:55.465878 32134 log.cc:1196] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/63403f5adb654ba580cfb59c16190df3
I20260809 22:36:55.466507 32134 ts_tablet_manager.cc:1939] T 63403f5adb654ba580cfb59c16190df3 P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting consensus metadata
I20260809 22:36:55.468475 31520 catalog_manager.cc:4841] TS f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919): tablet 63403f5adb654ba580cfb59c16190df3 (table test-table [id=9bedf10fb3f74a11ae6906b2a55185e1]) successfully deleted
I20260809 22:36:55.582101 32086 raft_consensus.cc:482] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:55.582387 32086 raft_consensus.cc:504] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.583297 32086 leader_election.cc:288] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:55.584116 32009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6e8be28c36400085db4ed408d8eee5" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" is_pre_election: true
I20260809 22:36:55.584323 31712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6e8be28c36400085db4ed408d8eee5" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" is_pre_election: true
I20260809 22:36:55.584668 31712 raft_consensus.cc:2454] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 0.
I20260809 22:36:55.584515 32009 raft_consensus.cc:2454] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 0.
I20260809 22:36:55.585224 31783 leader_election.cc:302] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:55.585558 32086 raft_consensus.cc:2781] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:55.585752 32086 raft_consensus.cc:482] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:55.585872 32086 raft_consensus.cc:3037] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:55.589587 32086 raft_consensus.cc:504] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.590451 32086 leader_election.cc:288] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 election: Requested vote from peers fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:55.591127 32009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6e8be28c36400085db4ed408d8eee5" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
I20260809 22:36:55.591388 32009 raft_consensus.cc:3037] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:55.591320 31712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6e8be28c36400085db4ed408d8eee5" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6fbd29677f44f99f7f61154e1d38c7"
I20260809 22:36:55.591586 31712 raft_consensus.cc:3037] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:55.594717 32009 raft_consensus.cc:2454] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 1.
I20260809 22:36:55.595335 31783 leader_election.cc:302] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, f6bbf9e6fd8c40299a50aafd34e0aa31; no voters:
I20260809 22:36:55.595400 31712 raft_consensus.cc:2454] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 1.
I20260809 22:36:55.595755 32086 raft_consensus.cc:2781] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:55.595955 32086 raft_consensus.cc:686] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 1 LEADER]: Becoming Leader. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Running, Role: LEADER
I20260809 22:36:55.596287 32086 consensus_queue.cc:237] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:55.599800 31533 catalog_manager.cc:5495] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ec428191bb749d78083ffb619ca913b (127.28.243.66). New cstate: current_term: 1 leader_uuid: "4ec428191bb749d78083ffb619ca913b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } health_report { overall_health: UNKNOWN } } }
I20260809 22:36:55.644345 32010 raft_consensus.cc:1261] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Refusing update from remote peer 4ec428191bb749d78083ffb619ca913b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:55.644424 31712 raft_consensus.cc:1261] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Refusing update from remote peer 4ec428191bb749d78083ffb619ca913b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:55.645326 32091 consensus_queue.cc:1035] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:55.645901 32091 consensus_queue.cc:1035] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:55.663636 32111 mvcc.cc:205] Tried to move back new op lower bound from 7316746304071303168 to 7316746303886041088. Current Snapshot: MvccSnapshot[applied={T|T < 7316746304071303168}]
W20260809 22:36:55.688961 32112 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 22:36:57.306092 31532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:57.306466 31532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:57.311280 31532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Sending DeleteTablet for 3 replicas of tablet 0c6e8be28c36400085db4ed408d8eee5
I20260809 22:36:57.312270 31683 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6e8be28c36400085db4ed408d8eee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:57 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:42034
I20260809 22:36:57.312270 31832 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6e8be28c36400085db4ed408d8eee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:57 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:49004
I20260809 22:36:57.312346 31986 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c6e8be28c36400085db4ed408d8eee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:57 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:59802
I20260809 22:36:57.326556 32167 tablet_replica.cc:330] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: stopping tablet replica
I20260809 22:36:57.326939 32167 raft_consensus.cc:2229] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:36:57.327472 32167 raft_consensus.cc:2258] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:57.328128 32169 tablet_replica.cc:330] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: stopping tablet replica
I20260809 22:36:57.328516 32169 raft_consensus.cc:2229] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:57.328923 32169 raft_consensus.cc:2258] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:57.329229 32167 ts_tablet_manager.cc:1905] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:57.330332 32169 ts_tablet_manager.cc:1905] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:57.338892 32169 ts_tablet_manager.cc:1918] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 22:36:57.339190 32169 log.cc:1196] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/0c6e8be28c36400085db4ed408d8eee5
I20260809 22:36:57.339793 32169 ts_tablet_manager.cc:1939] T 0c6e8be28c36400085db4ed408d8eee5 P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting consensus metadata
I20260809 22:36:57.339957 32168 tablet_replica.cc:330] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: stopping tablet replica
I20260809 22:36:57.340193 32167 ts_tablet_manager.cc:1918] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 22:36:57.340201 32168 raft_consensus.cc:2229] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:57.341208 32167 log.cc:1196] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/0c6e8be28c36400085db4ed408d8eee5
I20260809 22:36:57.341243 32168 raft_consensus.cc:2258] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:57.341614 31520 catalog_manager.cc:4841] TS f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919): tablet 0c6e8be28c36400085db4ed408d8eee5 (table test-table [id=c8fa89b7146b4b77985f88629cba6cbc]) successfully deleted
I20260809 22:36:57.341902 32167 ts_tablet_manager.cc:1939] T 0c6e8be28c36400085db4ed408d8eee5 P 4ec428191bb749d78083ffb619ca913b: Deleting consensus metadata
I20260809 22:36:57.342442 32168 ts_tablet_manager.cc:1905] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:57.343379 31517 catalog_manager.cc:4841] TS 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401): tablet 0c6e8be28c36400085db4ed408d8eee5 (table test-table [id=c8fa89b7146b4b77985f88629cba6cbc]) successfully deleted
I20260809 22:36:57.347949 32168 ts_tablet_manager.cc:1918] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 22:36:57.348100 32168 log.cc:1196] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/0c6e8be28c36400085db4ed408d8eee5
I20260809 22:36:57.348536 32168 ts_tablet_manager.cc:1939] T 0c6e8be28c36400085db4ed408d8eee5 P fa6fbd29677f44f99f7f61154e1d38c7: Deleting consensus metadata
I20260809 22:36:57.349932 31520 catalog_manager.cc:4841] TS fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685): tablet 0c6e8be28c36400085db4ed408d8eee5 (table test-table [id=c8fa89b7146b4b77985f88629cba6cbc]) successfully deleted
Aug 09 22:36:57 dist-test-slave-rgb1 krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 22:36:57 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:36:57.387843 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:57.371254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.28.243.126:41925)
0809 22:36:57.372308 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:57.372319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:57.372343 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:36:57.372395 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:57.372405 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:57.372466 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:57.372768 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:57.374136 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.374952 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.376276 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.376805 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.376949 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:57.381910 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:57.381916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:57.381919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:57.381965 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:57.384988 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:57.385721 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:57.385729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:57.385737 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:57.385870 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:57.386281 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:57.386286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:57.386289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:57.386598 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:57.386872 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:57.387174 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 22:36:57.387425 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":626,"threads_started":1}
I20260809 22:36:57.392490 31532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:36:57.393301 31532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:57.403128 31832 tablet_service.cc:1467] Processing CreateTablet for tablet d0b96acd7f3d41acaee16e0228e0668b (DEFAULT_TABLE table=test-table [id=914cb6ff4a9c4472aec6d0371506dc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:57.403671 31832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b96acd7f3d41acaee16e0228e0668b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:57.403700 31683 tablet_service.cc:1467] Processing CreateTablet for tablet d0b96acd7f3d41acaee16e0228e0668b (DEFAULT_TABLE table=test-table [id=914cb6ff4a9c4472aec6d0371506dc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:57.404114 31683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b96acd7f3d41acaee16e0228e0668b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:57.404289 31986 tablet_service.cc:1467] Processing CreateTablet for tablet d0b96acd7f3d41acaee16e0228e0668b (DEFAULT_TABLE table=test-table [id=914cb6ff4a9c4472aec6d0371506dc88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:57.404819 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b96acd7f3d41acaee16e0228e0668b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:57.416916 32182 tablet_bootstrap.cc:492] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: Bootstrap starting.
I20260809 22:36:57.418836 32182 tablet_bootstrap.cc:654] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:57.418902 32180 tablet_bootstrap.cc:492] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: Bootstrap starting.
I20260809 22:36:57.418942 32181 tablet_bootstrap.cc:492] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: Bootstrap starting.
I20260809 22:36:57.421166 32181 tablet_bootstrap.cc:654] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:57.421443 32180 tablet_bootstrap.cc:654] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:57.421995 32182 tablet_bootstrap.cc:492] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: No bootstrap required, opened a new log
I20260809 22:36:57.422191 32182 ts_tablet_manager.cc:1397] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260809 22:36:57.423225 32182 raft_consensus.cc:348] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.423595 32182 raft_consensus.cc:374] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:57.423961 32182 raft_consensus.cc:729] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bbf9e6fd8c40299a50aafd34e0aa31, State: Initialized, Role: FOLLOWER
I20260809 22:36:57.424324 32182 consensus_queue.cc:260] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.424412 32181 tablet_bootstrap.cc:492] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: No bootstrap required, opened a new log
I20260809 22:36:57.424778 32181 ts_tablet_manager.cc:1397] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 22:36:57.425899 32181 raft_consensus.cc:348] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.426227 32181 raft_consensus.cc:374] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:57.426348 32181 raft_consensus.cc:729] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6fbd29677f44f99f7f61154e1d38c7, State: Initialized, Role: FOLLOWER
I20260809 22:36:57.426726 32181 consensus_queue.cc:260] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.427575 32181 ts_tablet_manager.cc:1428] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 22:36:57.429210 32182 ts_tablet_manager.cc:1428] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260809 22:36:57.430092 32180 tablet_bootstrap.cc:492] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: No bootstrap required, opened a new log
I20260809 22:36:57.430272 32180 ts_tablet_manager.cc:1397] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260809 22:36:57.431079 32180 raft_consensus.cc:348] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.431376 32180 raft_consensus.cc:374] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:57.431520 32180 raft_consensus.cc:729] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Initialized, Role: FOLLOWER
I20260809 22:36:57.431780 32180 consensus_queue.cc:260] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.432509 32180 ts_tablet_manager.cc:1428] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260809 22:36:57.504523 32088 raft_consensus.cc:482] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:57.504793 32088 raft_consensus.cc:504] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.505611 32088 leader_election.cc:288] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:57.506274 32009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b96acd7f3d41acaee16e0228e0668b" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" is_pre_election: true
I20260809 22:36:57.506413 31856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b96acd7f3d41acaee16e0228e0668b" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec428191bb749d78083ffb619ca913b" is_pre_election: true
I20260809 22:36:57.506626 32009 raft_consensus.cc:2454] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 0.
I20260809 22:36:57.506701 31856 raft_consensus.cc:2454] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 0.
I20260809 22:36:57.507207 31633 leader_election.cc:302] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6bbf9e6fd8c40299a50aafd34e0aa31, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:57.507573 32088 raft_consensus.cc:2781] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:57.507731 32088 raft_consensus.cc:482] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:57.507849 32088 raft_consensus.cc:3037] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:57.510152 32088 raft_consensus.cc:504] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.510835 32088 leader_election.cc:288] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 election: Requested vote from peers 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:57.511420 31856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b96acd7f3d41acaee16e0228e0668b" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ec428191bb749d78083ffb619ca913b"
I20260809 22:36:57.511389 32009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b96acd7f3d41acaee16e0228e0668b" candidate_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
I20260809 22:36:57.511679 32009 raft_consensus.cc:3037] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:57.511690 31856 raft_consensus.cc:3037] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:57.513950 31856 raft_consensus.cc:2454] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 1.
I20260809 22:36:57.514461 31629 leader_election.cc:302] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:57.514776 32088 raft_consensus.cc:2781] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:57.514914 32009 raft_consensus.cc:2454] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa6fbd29677f44f99f7f61154e1d38c7 in term 1.
I20260809 22:36:57.515007 32088 raft_consensus.cc:686] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 LEADER]: Becoming Leader. State: Replica: fa6fbd29677f44f99f7f61154e1d38c7, State: Running, Role: LEADER
I20260809 22:36:57.515295 32088 consensus_queue.cc:237] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.518685 31533 catalog_manager.cc:5495] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } health_report { overall_health: UNKNOWN } } }
I20260809 22:36:57.563717 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:57.557895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.28.243.65:35685)
0809 22:36:57.558605 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:57.558610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:57.558630 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:36:57.558684 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:57.558688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:57.558751 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:57.558840 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:57.559318 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.560208 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.561789 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.562284 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.562385 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:57.562994 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:57.563116 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:57.563200 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:57.563362 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 22:36:57.563428 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":400,"threads_started":1}
I20260809 22:36:57.580333 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:57.574709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:54613 (local address 127.28.243.65:35685)
0809 22:36:57.574905 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:57.574910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:57.575513 (+ 603us) server_negotiation.cc:407] Connection header received
0809 22:36:57.575706 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:57.575717 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:57.575773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:57.575866 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:57.576327 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.577162 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.578605 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.579023 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.579152 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:57.579556 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:57.579654 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:57.579752 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:57.579884 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 22:36:57.579952 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260809 22:36:57.580384 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:57.574722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:35685 (local address 127.28.243.65:54613)
0809 22:36:57.575396 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0809 22:36:57.575412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:36:57.575587 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:36:57.575887 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:36:57.575892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:36:57.575902 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:36:57.576173 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:57.576182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.577287 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.577295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:57.578474 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:36:57.578487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.579249 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.579254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:36:57.579363 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:57.579397 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:36:57.579751 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:36:57.579764 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 22:36:57.579856 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 22:36:57.579993 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":22,"thread_start_us":405,"threads_started":1}
I20260809 22:36:57.589632 31856 raft_consensus.cc:1261] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Refusing update from remote peer fa6fbd29677f44f99f7f61154e1d38c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:57.589671 32009 raft_consensus.cc:1261] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Refusing update from remote peer fa6fbd29677f44f99f7f61154e1d38c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:57.590245 32152 consensus_queue.cc:1035] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:57.590636 32088 consensus_queue.cc:1035] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:57.641530 31532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:57.641798 31532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:57.645825 31532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Sending DeleteTablet for 3 replicas of tablet d0b96acd7f3d41acaee16e0228e0668b
I20260809 22:36:57.646819 31682 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b96acd7f3d41acaee16e0228e0668b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:57 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:42034
I20260809 22:36:57.647241 31832 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b96acd7f3d41acaee16e0228e0668b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:57 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:49004
I20260809 22:36:57.647646 31986 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b96acd7f3d41acaee16e0228e0668b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:57 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:59802
I20260809 22:36:57.647949 32169 tablet_replica.cc:330] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: stopping tablet replica
I20260809 22:36:57.647959 32168 tablet_replica.cc:330] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: stopping tablet replica
I20260809 22:36:57.648180 32168 raft_consensus.cc:2229] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:36:57.648175 32169 raft_consensus.cc:2229] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:57.648409 32169 raft_consensus.cc:2258] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:57.648496 31532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 22:36:57.648916 32168 raft_consensus.cc:2258] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:57.649243 32167 tablet_replica.cc:330] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: stopping tablet replica
I20260809 22:36:57.649485 32167 raft_consensus.cc:2229] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 22:36:57.649504 31532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:36:57.649689 32167 raft_consensus.cc:2258] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:57.650861 32168 ts_tablet_manager.cc:1905] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:57.651041 32169 ts_tablet_manager.cc:1905] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:57.651324 32167 ts_tablet_manager.cc:1905] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:57.657761 31832 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe21e3b455b40d3bf334da773b2562a (DEFAULT_TABLE table=test-table [id=7d2b6000a13444d9afbff0c6fae2d4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:57.657744 31682 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe21e3b455b40d3bf334da773b2562a (DEFAULT_TABLE table=test-table [id=7d2b6000a13444d9afbff0c6fae2d4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:57.658257 31682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe21e3b455b40d3bf334da773b2562a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:57.658283 31832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe21e3b455b40d3bf334da773b2562a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:57.658583 32167 ts_tablet_manager.cc:1918] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:57.658576 32169 ts_tablet_manager.cc:1918] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:57.658764 32167 log.cc:1196] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/d0b96acd7f3d41acaee16e0228e0668b
I20260809 22:36:57.658775 32169 log.cc:1196] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/d0b96acd7f3d41acaee16e0228e0668b
I20260809 22:36:57.659274 32169 ts_tablet_manager.cc:1939] T d0b96acd7f3d41acaee16e0228e0668b P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting consensus metadata
I20260809 22:36:57.659298 32167 ts_tablet_manager.cc:1939] T d0b96acd7f3d41acaee16e0228e0668b P 4ec428191bb749d78083ffb619ca913b: Deleting consensus metadata
I20260809 22:36:57.660542 31517 catalog_manager.cc:4841] TS 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401): tablet d0b96acd7f3d41acaee16e0228e0668b (table test-table [id=914cb6ff4a9c4472aec6d0371506dc88]) successfully deleted
I20260809 22:36:57.664990 31520 catalog_manager.cc:4841] TS f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919): tablet d0b96acd7f3d41acaee16e0228e0668b (table test-table [id=914cb6ff4a9c4472aec6d0371506dc88]) successfully deleted
I20260809 22:36:57.665592 32180 tablet_bootstrap.cc:492] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: Bootstrap starting.
I20260809 22:36:57.665510 31986 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe21e3b455b40d3bf334da773b2562a (DEFAULT_TABLE table=test-table [id=7d2b6000a13444d9afbff0c6fae2d4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:36:57.665953 31986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe21e3b455b40d3bf334da773b2562a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:36:57.666484 32168 ts_tablet_manager.cc:1918] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:36:57.666779 32168 log.cc:1196] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/d0b96acd7f3d41acaee16e0228e0668b
I20260809 22:36:57.667440 32180 tablet_bootstrap.cc:654] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:57.667420 32168 ts_tablet_manager.cc:1939] T d0b96acd7f3d41acaee16e0228e0668b P fa6fbd29677f44f99f7f61154e1d38c7: Deleting consensus metadata
I20260809 22:36:57.669842 32180 tablet_bootstrap.cc:492] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: No bootstrap required, opened a new log
I20260809 22:36:57.670054 32180 ts_tablet_manager.cc:1397] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260809 22:36:57.670848 32180 raft_consensus.cc:348] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.671193 32180 raft_consensus.cc:374] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:57.671263 32180 raft_consensus.cc:729] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Initialized, Role: FOLLOWER
I20260809 22:36:57.671693 32180 consensus_queue.cc:260] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.673208 32180 ts_tablet_manager.cc:1428] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 22:36:57.673631 32182 tablet_bootstrap.cc:492] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: Bootstrap starting.
I20260809 22:36:57.675568 32181 tablet_bootstrap.cc:492] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: Bootstrap starting.
I20260809 22:36:57.675731 32182 tablet_bootstrap.cc:654] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:57.677685 32181 tablet_bootstrap.cc:654] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: Neither blocks nor log segments found. Creating new log.
I20260809 22:36:57.678601 32182 tablet_bootstrap.cc:492] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: No bootstrap required, opened a new log
I20260809 22:36:57.678844 32182 ts_tablet_manager.cc:1397] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260809 22:36:57.679656 31520 catalog_manager.cc:4841] TS fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685): tablet d0b96acd7f3d41acaee16e0228e0668b (table test-table [id=914cb6ff4a9c4472aec6d0371506dc88]) successfully deleted
I20260809 22:36:57.681327 32181 tablet_bootstrap.cc:492] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: No bootstrap required, opened a new log
I20260809 22:36:57.681166 32182 raft_consensus.cc:348] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.681494 32182 raft_consensus.cc:374] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:57.681533 32181 ts_tablet_manager.cc:1397] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260809 22:36:57.681579 32182 raft_consensus.cc:729] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bbf9e6fd8c40299a50aafd34e0aa31, State: Initialized, Role: FOLLOWER
I20260809 22:36:57.681914 32182 consensus_queue.cc:260] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.682607 32181 raft_consensus.cc:348] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.682883 32181 raft_consensus.cc:374] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:36:57.682999 32181 raft_consensus.cc:729] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa6fbd29677f44f99f7f61154e1d38c7, State: Initialized, Role: FOLLOWER
I20260809 22:36:57.683265 32181 consensus_queue.cc:260] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.683765 32182 ts_tablet_manager.cc:1428] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260809 22:36:57.684407 32181 ts_tablet_manager.cc:1428] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 22:36:57.700047 32159 raft_consensus.cc:482] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:36:57.700335 32159 raft_consensus.cc:504] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.701164 32159 leader_election.cc:288] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:57.701730 31712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe21e3b455b40d3bf334da773b2562a" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" is_pre_election: true
I20260809 22:36:57.702144 31712 raft_consensus.cc:2454] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 0.
I20260809 22:36:57.702232 32010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe21e3b455b40d3bf334da773b2562a" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" is_pre_election: true
I20260809 22:36:57.702502 32010 raft_consensus.cc:2454] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 0.
I20260809 22:36:57.702946 31783 leader_election.cc:302] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:57.703282 32159 raft_consensus.cc:2781] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:36:57.703416 32159 raft_consensus.cc:482] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:36:57.703532 32159 raft_consensus.cc:3037] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:57.706633 32159 raft_consensus.cc:504] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.707350 32159 leader_election.cc:288] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 election: Requested vote from peers fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685), f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919)
I20260809 22:36:57.707840 31712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe21e3b455b40d3bf334da773b2562a" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa6fbd29677f44f99f7f61154e1d38c7"
I20260809 22:36:57.708137 31712 raft_consensus.cc:3037] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:57.710584 31712 raft_consensus.cc:2454] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 1.
I20260809 22:36:57.710999 32010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe21e3b455b40d3bf334da773b2562a" candidate_uuid: "4ec428191bb749d78083ffb619ca913b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31"
I20260809 22:36:57.711155 31783 leader_election.cc:302] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ec428191bb749d78083ffb619ca913b, fa6fbd29677f44f99f7f61154e1d38c7; no voters:
I20260809 22:36:57.711282 32010 raft_consensus.cc:3037] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:36:57.711510 32159 raft_consensus.cc:2781] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:36:57.711679 32159 raft_consensus.cc:686] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 1 LEADER]: Becoming Leader. State: Replica: 4ec428191bb749d78083ffb619ca913b, State: Running, Role: LEADER
I20260809 22:36:57.712031 32159 consensus_queue.cc:237] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } }
I20260809 22:36:57.715655 32010 raft_consensus.cc:2454] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ec428191bb749d78083ffb619ca913b in term 1.
I20260809 22:36:57.722378 31533 catalog_manager.cc:5495] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ec428191bb749d78083ffb619ca913b (127.28.243.66). New cstate: current_term: 1 leader_uuid: "4ec428191bb749d78083ffb619ca913b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ec428191bb749d78083ffb619ca913b" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 } health_report { overall_health: UNKNOWN } } }
I20260809 22:36:57.751163 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:57.744243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.28.243.66:33401)
0809 22:36:57.745077 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:57.745091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:57.745112 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:36:57.745169 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:57.745172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:57.745236 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:57.745322 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:36:57.745828 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.746789 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.749073 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:57.749583 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:57.749730 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:57.750257 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:36:57.750382 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:36:57.750458 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:57.750623 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 22:36:57.750697 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":548,"threads_started":1}
I20260809 22:36:57.762940 32010 raft_consensus.cc:1261] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Refusing update from remote peer 4ec428191bb749d78083ffb619ca913b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:57.763185 31712 raft_consensus.cc:1261] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Refusing update from remote peer 4ec428191bb749d78083ffb619ca913b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:36:57.763862 32159 consensus_queue.cc:1035] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bbf9e6fd8c40299a50aafd34e0aa31" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:36:57.764124 32086 consensus_queue.cc:1035] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa6fbd29677f44f99f7f61154e1d38c7" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 35685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 22:36:57.792224 32149 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 22:36:59.418881 31533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:59.419134 31533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:36:59.424273 31533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9ce188d68ed46b988934064a0259f6a: Sending DeleteTablet for 3 replicas of tablet 9fe21e3b455b40d3bf334da773b2562a
I20260809 22:36:59.425621 31986 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe21e3b455b40d3bf334da773b2562a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:59 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:59802
I20260809 22:36:59.425618 31832 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe21e3b455b40d3bf334da773b2562a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:59 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:49004
I20260809 22:36:59.425757 31682 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe21e3b455b40d3bf334da773b2562a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:36:59 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:42034
I20260809 22:36:59.426488 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31613
I20260809 22:36:59.441951 32233 tablet_replica.cc:330] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: stopping tablet replica
I20260809 22:36:59.443011 32234 tablet_replica.cc:330] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: stopping tablet replica
I20260809 22:36:59.443305 32234 raft_consensus.cc:2229] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:59.443709 32233 raft_consensus.cc:2229] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:36:59.444449 32234 raft_consensus.cc:2258] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:59.444199 32233 raft_consensus.cc:2258] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:59.444654 32232 tablet_replica.cc:330] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: stopping tablet replica
I20260809 22:36:59.445518 32232 raft_consensus.cc:2229] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:36:59.445636 32234 ts_tablet_manager.cc:1905] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:59.445784 32232 raft_consensus.cc:2258] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:36:59.446095 32233 ts_tablet_manager.cc:1905] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:59.446794 32232 ts_tablet_manager.cc:1905] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:36:59.453971 32233 ts_tablet_manager.cc:1918] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 22:36:59.453953 32234 ts_tablet_manager.cc:1918] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 22:36:59.454123 32234 log.cc:1196] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/9fe21e3b455b40d3bf334da773b2562a
I20260809 22:36:59.454149 32233 log.cc:1196] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/9fe21e3b455b40d3bf334da773b2562a
I20260809 22:36:59.454566 32233 ts_tablet_manager.cc:1939] T 9fe21e3b455b40d3bf334da773b2562a P 4ec428191bb749d78083ffb619ca913b: Deleting consensus metadata
I20260809 22:36:59.454589 32232 ts_tablet_manager.cc:1918] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 22:36:59.454656 32234 ts_tablet_manager.cc:1939] T 9fe21e3b455b40d3bf334da773b2562a P fa6fbd29677f44f99f7f61154e1d38c7: Deleting consensus metadata
I20260809 22:36:59.454738 32232 log.cc:1196] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/9fe21e3b455b40d3bf334da773b2562a
I20260809 22:36:59.455247 32232 ts_tablet_manager.cc:1939] T 9fe21e3b455b40d3bf334da773b2562a P f6bbf9e6fd8c40299a50aafd34e0aa31: Deleting consensus metadata
I20260809 22:36:59.455760 31517 catalog_manager.cc:4841] TS 4ec428191bb749d78083ffb619ca913b (127.28.243.66:33401): tablet 9fe21e3b455b40d3bf334da773b2562a (table test-table [id=7d2b6000a13444d9afbff0c6fae2d4e9]) successfully deleted
I20260809 22:36:59.455857 31520 catalog_manager.cc:4841] TS fa6fbd29677f44f99f7f61154e1d38c7 (127.28.243.65:35685): tablet 9fe21e3b455b40d3bf334da773b2562a (table test-table [id=7d2b6000a13444d9afbff0c6fae2d4e9]) successfully deleted
I20260809 22:36:59.456365 31520 catalog_manager.cc:4841] TS f6bbf9e6fd8c40299a50aafd34e0aa31 (127.28.243.67:36919): tablet 9fe21e3b455b40d3bf334da773b2562a (table test-table [id=7d2b6000a13444d9afbff0c6fae2d4e9]) successfully deleted
Aug 09 22:36:59 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:36:59.468178 32236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:59.431530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.28.243.65:35685)
0809 22:36:59.443869 (+ 12339us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:59.443875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:59.443906 (+ 31us) server_negotiation.cc:407] Connection header received
0809 22:36:59.444212 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:59.444222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:59.444332 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:59.444660 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:59.447281 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:59.448658 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:59.451758 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:59.452491 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:59.453332 (+ 841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:59.460306 (+ 6974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:59.460323 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:59.460339 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:59.460384 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:59.464546 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:59.465596 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:59.465604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:59.465606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:59.465799 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:59.466318 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:59.466322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:59.466324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:59.466805 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:59.467126 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:59.467482 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 22:36:59.467761 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12198,"thread_start_us":12028,"threads_started":1}
W20260809 22:36:59.469117 31754 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42046
W20260809 22:36:59.469859 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31613: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:36:59.470203 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 31613
I20260809 22:36:59.500298 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31764
Aug 09 22:36:59 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:36:59.518033 32237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:59.501232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.28.243.66:33401)
0809 22:36:59.501729 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:59.501734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:59.501752 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:36:59.501809 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:59.501813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:59.501877 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:59.502039 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:59.503360 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:59.504126 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:59.505605 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:59.506107 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:59.506215 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:59.510996 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:59.511017 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:59.511039 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:59.511085 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:59.515075 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:59.515818 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:59.515830 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:59.515836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:59.515992 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:59.516324 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:59.516329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:59.516333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:59.516825 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:59.517120 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:59.517431 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 22:36:59.517669 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
W20260809 22:36:59.518854 31903 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49036
W20260809 22:36:59.519538 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31764: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:36:59.519718 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 31764
I20260809 22:36:59.546478 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31912
Aug 09 22:36:59 dist-test-slave-rgb1 krb5kdc[31485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315017, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:36:59.565413 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:36:59.547402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.28.243.67:36919)
0809 22:36:59.548287 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0809 22:36:59.548292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:36:59.548311 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:36:59.548372 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:36:59.548376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:36:59.548515 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:36:59.548744 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:36:59.550341 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:59.551172 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:59.552507 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:36:59.553059 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:36:59.553212 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:36:59.558709 (+ 5497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:36:59.558728 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:36:59.558745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:36:59.558792 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:36:59.562299 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:59.563113 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:59.563122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:59.563124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:59.563276 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:36:59.563693 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:36:59.563699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:36:59.563702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:36:59.564106 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:36:59.564443 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0809 22:36:59.564804 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 22:36:59.565068 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":497,"threads_started":1}
W20260809 22:36:59.566139 32052 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59812
W20260809 22:36:59.566864 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31912: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:36:59.567132 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 31912
I20260809 22:36:59.588802 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 31500
I20260809 22:36:59.590277 31580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:36:59.711946 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 31500
2026-08-09T22:36:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8899 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:36:59 dist-test-slave-rgb1 krb5kdc[32242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:36:59 dist-test-slave-rgb1 krb5kdc[32242](info): set up 2 sockets
Aug 09 22:36:59 dist-test-slave-rgb1 krb5kdc[32242](info): commencing operation
krb5kdc: starting...
W20260809 22:37:00.934581 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:01 dist-test-slave-rgb1 krb5kdc[32242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:01.190466 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34773
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:34773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:01.549304 32255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:01.549630 32255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:01.549726 32255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:01.557889 32255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:01.558025 32255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:01.558075 32255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:01.558126 32255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:01.558173 32255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:01.568797 32255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:34773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:01.570150 32255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:01.571986 32255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:01.587627 32260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:01.587534 32261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:01.587662 32263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:01.587988 32255 server_base.cc:1034] running on GCE node
I20260809 22:37:01.589114 32255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:01.590752 32255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:01.592046 32255 hybrid_clock.cc:648] HybridClock initialized: now 1786315021591976 us; error 49 us; skew 500 ppm
I20260809 22:37:01.592536 32255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:01 dist-test-slave-rgb1 krb5kdc[32242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:01.596338 32255 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:01.598235 32255 webserver.cc:460] Webserver started at http://127.28.243.126:34111/ using document root <none> and password file <none>
I20260809 22:37:01.599119 32255 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:01.599291 32255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:01.599664 32255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:01.603353 32255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "3be445709be946e0bd765373f5300ffd"
format_stamp: "Formatted at 2026-08-09 22:37:01 on dist-test-slave-rgb1"
server_key: "392326bce67df3aa6a1d4f560276fdee"
server_key_iv: "38ef58ecd3a23778738bfeca66508b79"
server_key_version: "encryptionkey@0"
I20260809 22:37:01.604285 32255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "3be445709be946e0bd765373f5300ffd"
format_stamp: "Formatted at 2026-08-09 22:37:01 on dist-test-slave-rgb1"
server_key: "392326bce67df3aa6a1d4f560276fdee"
server_key_iv: "38ef58ecd3a23778738bfeca66508b79"
server_key_version: "encryptionkey@0"
I20260809 22:37:01.610538 32255 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260809 22:37:01.614883 32271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:01.616998 32255 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 22:37:01.617276 32255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "3be445709be946e0bd765373f5300ffd"
format_stamp: "Formatted at 2026-08-09 22:37:01 on dist-test-slave-rgb1"
server_key: "392326bce67df3aa6a1d4f560276fdee"
server_key_iv: "38ef58ecd3a23778738bfeca66508b79"
server_key_version: "encryptionkey@0"
I20260809 22:37:01.617586 32255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:01.637535 32255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:01.642717 32255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:01.643006 32255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:01.666556 32255 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:34773
I20260809 22:37:01.666594 32327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:34773 every 8 connection(s)
I20260809 22:37:01.668181 32255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:01.673270 32328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:01.676838 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 32255
I20260809 22:37:01.677095 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:01.677793 29645 external_mini_cluster.cc:1428] Setting key 13090c96cc57d9804037657c285cd7c4
I20260809 22:37:01.689371 32328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd: Bootstrap starting.
I20260809 22:37:01.693442 32328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:01.694581 32328 log.cc:824] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd: Log is configured to *not* fsync() on all Append() calls
Aug 09 22:37:01 dist-test-slave-rgb1 krb5kdc[32242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:01.699084 32328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd: No bootstrap required, opened a new log
I20260809 22:37:01.705789 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:01.680202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.28.243.126:34773)
0809 22:37:01.681176 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:01.681195 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:01.681255 (+ 60us) server_negotiation.cc:407] Connection header received
0809 22:37:01.684052 (+ 2797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:01.684091 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:01.684768 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:01.685305 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:01.687316 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:01.688816 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:01.690505 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:01.691236 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:01.691481 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:01.696023 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:01.696047 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:01.696069 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:01.696128 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:01.700236 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:01.701330 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:01.701338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:01.701348 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:01.701566 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:01.702058 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:01.702064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:01.702067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:01.702901 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:01.703242 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:01.703729 (+ 487us) server_negotiation.cc:299] Negotiation successful
0809 22:37:01.704228 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":414,"threads_started":1}
I20260809 22:37:01.709714 32328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be445709be946e0bd765373f5300ffd" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34773 } }
I20260809 22:37:01.710078 32328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:01.710155 32328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be445709be946e0bd765373f5300ffd, State: Initialized, Role: FOLLOWER
I20260809 22:37:01.710647 32328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be445709be946e0bd765373f5300ffd" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34773 } }
I20260809 22:37:01.710850 32328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:01.710950 32328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:01.711082 32328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:01.713488 32328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be445709be946e0bd765373f5300ffd" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34773 } }
I20260809 22:37:01.714123 32328 leader_election.cc:302] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3be445709be946e0bd765373f5300ffd; no voters:
I20260809 22:37:01.714812 32328 leader_election.cc:288] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:01.715058 32334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:01.715863 32334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [term 1 LEADER]: Becoming Leader. State: Replica: 3be445709be946e0bd765373f5300ffd, State: Running, Role: LEADER
I20260809 22:37:01.716394 32334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be445709be946e0bd765373f5300ffd" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34773 } }
I20260809 22:37:01.717034 32328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:01.719655 32335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3be445709be946e0bd765373f5300ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be445709be946e0bd765373f5300ffd" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34773 } } }
I20260809 22:37:01.720148 32335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:01.720235 32336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3be445709be946e0bd765373f5300ffd. Latest consensus state: current_term: 1 leader_uuid: "3be445709be946e0bd765373f5300ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3be445709be946e0bd765373f5300ffd" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34773 } } }
I20260809 22:37:01.720530 32336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:01.723419 32343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:01.729286 32343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:01.739522 32343 catalog_manager.cc:1347] Generated new cluster ID: 06de111d17be41a59bcffe39efa4b5a8
I20260809 22:37:01.739670 32343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:01.779445 32343 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:01.781169 32343 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:01.794186 32343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3be445709be946e0bd765373f5300ffd: Generated new TSK 0
I20260809 22:37:01.795296 32343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:37:01.866042 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34773
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:02.251698 32355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:02.252012 32355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:02.252177 32355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:02.259575 32355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:02.259713 32355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:02.259917 32355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:02.270069 32355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:02.271848 32355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:02.273377 32355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:02.287897 32361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:02.288064 32363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:02.288089 32360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:02.289032 32355 server_base.cc:1034] running on GCE node
I20260809 22:37:02.289791 32355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:02.290861 32355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:02.292163 32355 hybrid_clock.cc:648] HybridClock initialized: now 1786315022292103 us; error 48 us; skew 500 ppm
I20260809 22:37:02.292721 32355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:02 dist-test-slave-rgb1 krb5kdc[32242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:02.296490 32355 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:37:02.298784 32355 webserver.cc:460] Webserver started at http://127.28.243.65:35953/ using document root <none> and password file <none>
I20260809 22:37:02.299577 32355 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:02.299739 32355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:02.300076 32355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:02.303254 32355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "defbc5c9616344ac9ea7136c88538ee2"
format_stamp: "Formatted at 2026-08-09 22:37:02 on dist-test-slave-rgb1"
server_key: "477497cca3bfc63ddd1a0bb50d3793d7"
server_key_iv: "c74d31a8da37f74d75ca1a49a4f8932c"
server_key_version: "encryptionkey@0"
I20260809 22:37:02.304186 32355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "defbc5c9616344ac9ea7136c88538ee2"
format_stamp: "Formatted at 2026-08-09 22:37:02 on dist-test-slave-rgb1"
server_key: "477497cca3bfc63ddd1a0bb50d3793d7"
server_key_iv: "c74d31a8da37f74d75ca1a49a4f8932c"
server_key_version: "encryptionkey@0"
I20260809 22:37:02.310171 32355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 22:37:02.314277 32370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:02.315874 32355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260809 22:37:02.316083 32355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "defbc5c9616344ac9ea7136c88538ee2"
format_stamp: "Formatted at 2026-08-09 22:37:02 on dist-test-slave-rgb1"
server_key: "477497cca3bfc63ddd1a0bb50d3793d7"
server_key_iv: "c74d31a8da37f74d75ca1a49a4f8932c"
server_key_version: "encryptionkey@0"
I20260809 22:37:02.316293 32355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:02.343039 32355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:02.348112 32355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:02.348487 32355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:02.349684 32355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:02.351189 32355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:02.351269 32355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:02.351429 32355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:02.351500 32355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:02.393325 32355 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:35745
I20260809 22:37:02.393425 32495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:35745 every 8 connection(s)
I20260809 22:37:02.394673 32355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:02.396168 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 32355
I20260809 22:37:02.396437 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:02.397115 29645 external_mini_cluster.cc:1428] Setting key 6d5ebde68995ec17f730219f271db9fd
Aug 09 22:37:02 dist-test-slave-rgb1 krb5kdc[32242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:02.423487 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:02.398503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:35311 (local address 127.28.243.126:34773)
0809 22:37:02.399043 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:02.399053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:02.399859 (+ 806us) server_negotiation.cc:407] Connection header received
0809 22:37:02.403022 (+ 3163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:02.403027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:02.403102 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:02.403211 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:02.405684 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.406577 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.408805 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.409647 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.409825 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:02.414081 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:02.414093 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:02.414103 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:02.414158 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:02.418666 (+ 4508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:02.419687 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:02.419696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:02.419702 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:02.419912 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:02.420734 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:02.420739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:02.420741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:02.421227 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:02.421426 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:02.422907 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0809 22:37:02.423183 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:37:02.425330 32499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:02.398548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34773 (local address 127.28.243.65:35311)
0809 22:37:02.399675 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:02.399736 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:02.402663 (+ 2927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:02.403552 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:02.403567 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:02.404473 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:02.405420 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:02.405463 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.406776 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.406784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:02.408545 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:02.408628 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.409942 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.409949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:02.410170 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:02.411040 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:02.411064 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:02.413753 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:02.418873 (+ 5120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:02.418882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:02.418898 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:02.419422 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:02.420133 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:02.420141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:02.420149 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:02.420521 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:02.421483 (+ 962us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:02.421495 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:02.422097 (+ 602us) client_negotiation.cc:769] Sending connection context
0809 22:37:02.422565 (+ 468us) client_negotiation.cc:240] Negotiation successful
0809 22:37:02.423338 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":389,"threads_started":1}
I20260809 22:37:02.427173 32496 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34773
I20260809 22:37:02.427543 32496 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
I20260809 22:37:02.428517 32496 heartbeater.cc:511] Master 127.28.243.126:34773 requested a full tablet report, sending...
I20260809 22:37:02.431401 32290 ts_manager.cc:194] Registered new tserver with Master: defbc5c9616344ac9ea7136c88538ee2 (127.28.243.65:35745)
I20260809 22:37:02.433013 32290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:35311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:37:02.457453 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34773
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:02.813700 32502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:02.814023 32502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:02.814122 32502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:02.821983 32502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:02.822098 32502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:02.822271 32502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:02.833350 32502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:02.834801 32502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:02.836306 32502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:02.851433 32510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:02.851330 32508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:02.851471 32512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:02.852854 32502 server_base.cc:1034] running on GCE node
I20260809 22:37:02.853578 32502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:02.854602 32502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:02.855902 32502 hybrid_clock.cc:648] HybridClock initialized: now 1786315022855792 us; error 34 us; skew 500 ppm
I20260809 22:37:02.856298 32502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:02 dist-test-slave-rgb1 krb5kdc[32242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:02.860355 32502 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:37:02.862274 32502 webserver.cc:460] Webserver started at http://127.28.243.66:35925/ using document root <none> and password file <none>
I20260809 22:37:02.863059 32502 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:02.863235 32502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:02.863602 32502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:02.866953 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "1506e28b0667451ba4a3d189d594bb56"
format_stamp: "Formatted at 2026-08-09 22:37:02 on dist-test-slave-rgb1"
server_key: "bb2440d58bec2cc8cfff29c71aee68ad"
server_key_iv: "ecc6390b842cced10cb37d276994b845"
server_key_version: "encryptionkey@0"
I20260809 22:37:02.867900 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "1506e28b0667451ba4a3d189d594bb56"
format_stamp: "Formatted at 2026-08-09 22:37:02 on dist-test-slave-rgb1"
server_key: "bb2440d58bec2cc8cfff29c71aee68ad"
server_key_iv: "ecc6390b842cced10cb37d276994b845"
server_key_version: "encryptionkey@0"
I20260809 22:37:02.874156 32502 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 22:37:02.878088 32519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:02.879565 32502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:37:02.879774 32502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "1506e28b0667451ba4a3d189d594bb56"
format_stamp: "Formatted at 2026-08-09 22:37:02 on dist-test-slave-rgb1"
server_key: "bb2440d58bec2cc8cfff29c71aee68ad"
server_key_iv: "ecc6390b842cced10cb37d276994b845"
server_key_version: "encryptionkey@0"
I20260809 22:37:02.880030 32502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:02.903987 32502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:02.908921 32502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:02.909243 32502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:02.910356 32502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:02.911868 32502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:02.911988 32502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:02.912155 32502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:02.912235 32502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:02.952231 32502 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:34203
I20260809 22:37:02.952481 32639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:34203 every 8 connection(s)
I20260809 22:37:02.953787 32502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:02.962805 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 32502
I20260809 22:37:02.963063 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:02.963518 29645 external_mini_cluster.cc:1428] Setting key 910e6affa1c606e2e5d503ed30c44287
Aug 09 22:37:02 dist-test-slave-rgb1 krb5kdc[32242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:02.980667 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:02.956876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:57683 (local address 127.28.243.126:34773)
0809 22:37:02.957385 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:02.957390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:02.958634 (+ 1244us) server_negotiation.cc:407] Connection header received
0809 22:37:02.960967 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:02.960973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:02.961083 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:02.961222 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:02.962921 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.963883 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.966375 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.966879 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.967037 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:02.972359 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:02.972364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:02.972368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:02.972405 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:02.976363 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:02.977517 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:02.977527 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:02.977533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:02.977734 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:02.978289 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:02.978297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:02.978304 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:02.978716 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:02.978884 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:02.979794 (+ 910us) server_negotiation.cc:299] Negotiation successful
0809 22:37:02.980105 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":251,"threads_started":1}
I20260809 22:37:02.981475 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:02.957474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34773 (local address 127.28.243.66:57683)
0809 22:37:02.958422 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:02.958485 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:02.960641 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:02.961511 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:02.961521 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:02.962085 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:02.962703 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:02.962723 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.964100 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.964109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:02.966179 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:02.966195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:02.967164 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:02.967172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:02.967457 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:02.968294 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:02.968405 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:02.972019 (+ 3614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:02.976652 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:02.976758 (+ 106us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:02.976776 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:02.977335 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:02.977887 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:02.977892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:02.977894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:02.978149 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:02.978885 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:02.978893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:02.979372 (+ 479us) client_negotiation.cc:769] Sending connection context
0809 22:37:02.979649 (+ 277us) client_negotiation.cc:240] Negotiation successful
0809 22:37:02.980110 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":403,"threads_started":1}
I20260809 22:37:02.983212 32640 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34773
I20260809 22:37:02.983592 32640 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:02.984658 32640 heartbeater.cc:511] Master 127.28.243.126:34773 requested a full tablet report, sending...
I20260809 22:37:02.986784 32290 ts_manager.cc:194] Registered new tserver with Master: 1506e28b0667451ba4a3d189d594bb56 (127.28.243.66:34203)
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
I20260809 22:37:02.987789 32290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:57683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:37:03.020957 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34773
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:43237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:03.368647 32646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:03.368968 32646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:03.369148 32646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:03.377698 32646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:03.377830 32646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:03.378022 32646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:03.388499 32646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:43237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:03.389938 32646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:03.391350 32646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:03.409250 32654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:03.409513 32651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:03.410073 32652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:03.411733 32646 server_base.cc:1034] running on GCE node
I20260809 22:37:03.412520 32646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:03.413630 32646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:03.414909 32646 hybrid_clock.cc:648] HybridClock initialized: now 1786315023414829 us; error 74 us; skew 500 ppm
I20260809 22:37:03.415374 32646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:03 dist-test-slave-rgb1 krb5kdc[32242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:03.419893 32646 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:37:03.422209 32646 webserver.cc:460] Webserver started at http://127.28.243.67:42233/ using document root <none> and password file <none>
I20260809 22:37:03.423110 32646 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:03.423273 32646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:03.423631 32646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:03.426857 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "7cb8f2a06bf048d49b661b80c0216444"
format_stamp: "Formatted at 2026-08-09 22:37:03 on dist-test-slave-rgb1"
server_key: "0157ec7094b5d8d6e6e48aa4710f1b03"
server_key_iv: "b67320f12c661674919368bbdb65e71e"
server_key_version: "encryptionkey@0"
I20260809 22:37:03.427812 32646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "7cb8f2a06bf048d49b661b80c0216444"
format_stamp: "Formatted at 2026-08-09 22:37:03 on dist-test-slave-rgb1"
server_key: "0157ec7094b5d8d6e6e48aa4710f1b03"
server_key_iv: "b67320f12c661674919368bbdb65e71e"
server_key_version: "encryptionkey@0"
I20260809 22:37:03.433553 32646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 22:37:03.436722 32496 heartbeater.cc:503] Master 127.28.243.126:34773 was elected leader, sending a full tablet report...
I20260809 22:37:03.437799 32661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:03.439702 32646 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:37:03.439867 32646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "7cb8f2a06bf048d49b661b80c0216444"
format_stamp: "Formatted at 2026-08-09 22:37:03 on dist-test-slave-rgb1"
server_key: "0157ec7094b5d8d6e6e48aa4710f1b03"
server_key_iv: "b67320f12c661674919368bbdb65e71e"
server_key_version: "encryptionkey@0"
I20260809 22:37:03.440068 32646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:03.460187 32646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:03.465502 32646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:03.465781 32646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:03.466802 32646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:03.468298 32646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:03.468377 32646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:03.468480 32646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:03.468519 32646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:03.510522 32646 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:45549
I20260809 22:37:03.510574 314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:45549 every 8 connection(s)
I20260809 22:37:03.512019 32646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:03.515771 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 32646
I20260809 22:37:03.516021 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:03.516618 29645 external_mini_cluster.cc:1428] Setting key 2b7dc65abe9ff2fccccea08e5b253129
Aug 09 22:37:03 dist-test-slave-rgb1 krb5kdc[32242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:03.535873 317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:03.515007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:60109 (local address 127.28.243.126:34773)
0809 22:37:03.515527 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:03.515532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:03.516739 (+ 1207us) server_negotiation.cc:407] Connection header received
0809 22:37:03.519316 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:03.519322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:03.519440 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:03.519575 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:03.521657 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:03.522674 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:03.524323 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:03.524808 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:03.524950 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:03.528915 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:03.528921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:03.528925 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:03.528962 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:03.532211 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:03.533182 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:03.533190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:03.533194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:03.533330 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:03.533858 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:03.533865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:03.533868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:03.534179 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:03.534373 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:03.535181 (+ 808us) server_negotiation.cc:299] Negotiation successful
0809 22:37:03.535444 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":295,"threads_started":1}
I20260809 22:37:03.536995 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:03.515554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34773 (local address 127.28.243.67:60109)
0809 22:37:03.516482 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:03.516539 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:03.519007 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:03.519930 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:03.519939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:03.520692 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:03.521456 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:03.521489 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:03.522857 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:03.522861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:03.524164 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:03.524177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:03.525066 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:03.525074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:03.525319 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:03.526069 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:03.526097 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:03.528636 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:03.532391 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:03.532403 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:03.532423 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:03.533034 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:03.533470 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:03.533477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:03.533481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:03.533721 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:03.534331 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:03.534340 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:03.534745 (+ 405us) client_negotiation.cc:769] Sending connection context
0809 22:37:03.535046 (+ 301us) client_negotiation.cc:240] Negotiation successful
0809 22:37:03.535674 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":431,"threads_started":1}
I20260809 22:37:03.538743 315 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34773
I20260809 22:37:03.539114 315 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:03.539914 315 heartbeater.cc:511] Master 127.28.243.126:34773 requested a full tablet report, sending...
I20260809 22:37:03.541535 32290 ts_manager.cc:194] Registered new tserver with Master: 7cb8f2a06bf048d49b661b80c0216444 (127.28.243.67:45549)
I20260809 22:37:03.542382 32290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:60109
I20260809 22:37:03.545392 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 22:37:03.586196 317 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:03.583038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.28.243.126:34773)
0809 22:37:03.583238 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:03.583243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:03.583714 (+ 471us) server_negotiation.cc:407] Connection header received
0809 22:37:03.583847 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:03.583851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:03.583987 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:03.584093 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:03.585833 (+ 1740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55656: BlockingRecv error: recv got EOF from 127.0.0.1:55656 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 22:37:03.589947 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 32355
W20260809 22:37:03.593201 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 32355: Not authorized: Client connection negotiation failed: client connection to 127.28.243.65:35745: server requires authentication, but client does not have Kerberos credentials available
I20260809 22:37:03.593395 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 32355
I20260809 22:37:03.613266 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 32502
W20260809 22:37:03.616376 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 32502: Not authorized: Client connection negotiation failed: client connection to 127.28.243.66:34203: server requires authentication, but client does not have Kerberos credentials available
I20260809 22:37:03.616654 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 32502
W20260809 22:37:03.616724 328 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:03.614415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.28.243.66:34203)
0809 22:37:03.615012 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:03.615021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:03.615047 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:37:03.615111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:03.615117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:03.615340 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:03.615480 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:03.616410 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35316: BlockingRecv error: recv got EOF from 127.0.0.1:35316 (error 108)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
I20260809 22:37:03.635151 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 32646
W20260809 22:37:03.638722 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 32646: Not authorized: Client connection negotiation failed: client connection to 127.28.243.67:45549: server requires authentication, but client does not have Kerberos credentials available
I20260809 22:37:03.638890 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 32646
W20260809 22:37:03.639005 329 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:03.636343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.28.243.67:45549)
0809 22:37:03.637247 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:03.637254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:03.637289 (+ 35us) server_negotiation.cc:407] Connection header received
0809 22:37:03.637361 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:03.637368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:03.637617 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:03.637808 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:03.638649 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52438: BlockingRecv error: recv got EOF from 127.0.0.1:52438 (error 108)
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":611,"threads_started":1}
I20260809 22:37:03.657871 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 32255
I20260809 22:37:03.659416 32323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:03.765537 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 32255
2026-08-09T22:37:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4051 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:03 dist-test-slave-rgb1 krb5kdc[334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:03 dist-test-slave-rgb1 krb5kdc[334](info): set up 2 sockets
Aug 09 22:37:03 dist-test-slave-rgb1 krb5kdc[334](info): commencing operation
krb5kdc: starting...
W20260809 22:37:04.980168 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:05 dist-test-slave-rgb1 krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:05.238667 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:46203
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:46203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:05.606210 349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:05.606563 349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:05.606753 349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:05.614537 349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:05.614677 349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:05.614785 349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:05.614907 349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:05.614993 349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:05.625620 349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:46203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:05.627050 349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:05.628741 349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:05.641923 357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:05.641963 354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:05.641861 355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:05.642953 349 server_base.cc:1034] running on GCE node
I20260809 22:37:05.643925 349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:05.645658 349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:05.646978 349 hybrid_clock.cc:648] HybridClock initialized: now 1786315025646929 us; error 73 us; skew 500 ppm
I20260809 22:37:05.647396 349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:05 dist-test-slave-rgb1 krb5kdc[334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315025, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:05.655133 349 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:05.656940 349 webserver.cc:460] Webserver started at http://127.28.243.126:33701/ using document root <none> and password file <none>
I20260809 22:37:05.657694 349 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:05.657831 349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:05.658197 349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:05.661479 349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "81640776efaf4125be5886fe4291f631"
format_stamp: "Formatted at 2026-08-09 22:37:05 on dist-test-slave-rgb1"
server_key: "d1fd0bd4357f09ec96f615cbd7ea83f8"
server_key_iv: "ce252b54fd945c46b5a53d6f54da05fc"
server_key_version: "encryptionkey@0"
I20260809 22:37:05.662319 349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "81640776efaf4125be5886fe4291f631"
format_stamp: "Formatted at 2026-08-09 22:37:05 on dist-test-slave-rgb1"
server_key: "d1fd0bd4357f09ec96f615cbd7ea83f8"
server_key_iv: "ce252b54fd945c46b5a53d6f54da05fc"
server_key_version: "encryptionkey@0"
I20260809 22:37:05.668171 349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260809 22:37:05.672091 364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:05.673858 349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 22:37:05.674079 349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "81640776efaf4125be5886fe4291f631"
format_stamp: "Formatted at 2026-08-09 22:37:05 on dist-test-slave-rgb1"
server_key: "d1fd0bd4357f09ec96f615cbd7ea83f8"
server_key_iv: "ce252b54fd945c46b5a53d6f54da05fc"
server_key_version: "encryptionkey@0"
I20260809 22:37:05.674320 349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:05.702533 349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:05.707757 349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:05.708103 349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:05.730031 349 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:46203
I20260809 22:37:05.730084 420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:46203 every 8 connection(s)
I20260809 22:37:05.731590 349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:05.735273 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 349
I20260809 22:37:05.735561 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:05.735693 421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:05.736407 29645 external_mini_cluster.cc:1428] Setting key fbd721fe1f5523c6bcdc3fe1fdc0a9d2
I20260809 22:37:05.750551 421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631: Bootstrap starting.
Aug 09 22:37:05 dist-test-slave-rgb1 krb5kdc[334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:05.754593 421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:05.756119 421 log.cc:824] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:05.759268 421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631: No bootstrap required, opened a new log
I20260809 22:37:05.763191 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:05.738494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.28.243.126:46203)
0809 22:37:05.739553 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:05.739566 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:05.739608 (+ 42us) server_negotiation.cc:407] Connection header received
0809 22:37:05.741834 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:05.741859 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:05.742302 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:05.742656 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:05.744676 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:05.746221 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:05.748215 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:05.749105 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:05.749254 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:05.754888 (+ 5634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:05.754904 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:05.754922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:05.754970 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:05.758510 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:05.759308 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:05.759333 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:05.759343 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:05.759540 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:05.759944 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:05.759949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:05.759953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:05.760761 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:05.761105 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:05.761637 (+ 532us) server_negotiation.cc:299] Negotiation successful
0809 22:37:05.762098 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":450,"threads_started":1}
I20260809 22:37:05.767421 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:37:05.768456 421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81640776efaf4125be5886fe4291f631" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 46203 } }
I20260809 22:37:05.768934 421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:05.769037 421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81640776efaf4125be5886fe4291f631, State: Initialized, Role: FOLLOWER
I20260809 22:37:05.769713 421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81640776efaf4125be5886fe4291f631" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 46203 } }
I20260809 22:37:05.769960 421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:05.770105 421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:05.770274 421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:05.772830 421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81640776efaf4125be5886fe4291f631" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 46203 } }
I20260809 22:37:05.773418 421 leader_election.cc:302] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81640776efaf4125be5886fe4291f631; no voters:
I20260809 22:37:05.774094 421 leader_election.cc:288] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:05.774204 428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:05.775627 428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [term 1 LEADER]: Becoming Leader. State: Replica: 81640776efaf4125be5886fe4291f631, State: Running, Role: LEADER
I20260809 22:37:05.776273 428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81640776efaf4125be5886fe4291f631" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 46203 } }
I20260809 22:37:05.776810 421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:05.779927 429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81640776efaf4125be5886fe4291f631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81640776efaf4125be5886fe4291f631" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 46203 } } }
I20260809 22:37:05.780808 429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:05.779911 430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81640776efaf4125be5886fe4291f631. Latest consensus state: current_term: 1 leader_uuid: "81640776efaf4125be5886fe4291f631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81640776efaf4125be5886fe4291f631" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 46203 } } }
I20260809 22:37:05.782541 430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:05.782740 436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:05.790493 436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:05.802277 436 catalog_manager.cc:1347] Generated new cluster ID: c38de866168c411ba14ef078b6764b60
I20260809 22:37:05.802415 436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:05.832607 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:05.823974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.28.243.126:46203)
0809 22:37:05.824246 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:05.824255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:05.825215 (+ 960us) server_negotiation.cc:407] Connection header received
0809 22:37:05.825449 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:05.825454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:05.825524 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:05.825656 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:05.826726 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:05.827489 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:05.829111 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:05.829543 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:05.829694 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:05.830838 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:05.830843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:05.830847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:05.831667 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:05.831731 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:05.831759 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:05.831903 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:05.832035 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 22:37:05.832149 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 22:37:05.839932 436 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:05.841850 436 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:05.860111 436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81640776efaf4125be5886fe4291f631: Generated new TSK 0
I20260809 22:37:05.861373 436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:37:05.877149 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 349
I20260809 22:37:05.879290 416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:05.981050 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 349
2026-08-09T22:37:05Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2209 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:06 dist-test-slave-rgb1 krb5kdc[460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:06 dist-test-slave-rgb1 krb5kdc[460](info): set up 2 sockets
Aug 09 22:37:06 dist-test-slave-rgb1 krb5kdc[460](info): commencing operation
krb5kdc: starting...
W20260809 22:37:07.161708 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:07 dist-test-slave-rgb1 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:07.406252 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38467
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:07.760236 473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:07.760588 473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:07.760772 473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:07.768383 473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:07.768527 473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:07.768613 473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:07.768668 473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:07.768759 473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:07.779567 473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:38467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:07.781136 473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:07.782510 473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:07.795285 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:07.795395 479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:07.795392 480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:07.796756 473 server_base.cc:1034] running on GCE node
I20260809 22:37:07.797655 473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:07.799196 473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:07.800462 473 hybrid_clock.cc:648] HybridClock initialized: now 1786315027800405 us; error 70 us; skew 500 ppm
I20260809 22:37:07.800868 473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:07 dist-test-slave-rgb1 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315027, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:07.804198 473 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:07.805864 473 webserver.cc:460] Webserver started at http://127.28.243.126:35241/ using document root <none> and password file <none>
I20260809 22:37:07.806516 473 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:07.806624 473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:07.806876 473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:07.810048 473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "3cdb5bbe37f44e789cc15891fd47820e"
format_stamp: "Formatted at 2026-08-09 22:37:07 on dist-test-slave-rgb1"
server_key: "655d6a34827659aa151c16a658a228c9"
server_key_iv: "43b242ba1c1df3008e9bdeac811b1fe2"
server_key_version: "encryptionkey@0"
I20260809 22:37:07.810771 473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "3cdb5bbe37f44e789cc15891fd47820e"
format_stamp: "Formatted at 2026-08-09 22:37:07 on dist-test-slave-rgb1"
server_key: "655d6a34827659aa151c16a658a228c9"
server_key_iv: "43b242ba1c1df3008e9bdeac811b1fe2"
server_key_version: "encryptionkey@0"
I20260809 22:37:07.816170 473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 22:37:07.819839 490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:07.821297 473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:07.821484 473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "3cdb5bbe37f44e789cc15891fd47820e"
format_stamp: "Formatted at 2026-08-09 22:37:07 on dist-test-slave-rgb1"
server_key: "655d6a34827659aa151c16a658a228c9"
server_key_iv: "43b242ba1c1df3008e9bdeac811b1fe2"
server_key_version: "encryptionkey@0"
I20260809 22:37:07.821748 473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:07.868119 473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:07.873021 473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:07.873361 473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:07.893880 473 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:38467
I20260809 22:37:07.893925 545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:38467 every 8 connection(s)
I20260809 22:37:07.895349 473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:07.899427 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:07.901916 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 473
I20260809 22:37:07.902158 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:07.903086 29645 external_mini_cluster.cc:1428] Setting key 4f77401ea85c73803f363c8c728802e3
I20260809 22:37:07.914234 546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e: Bootstrap starting.
I20260809 22:37:07.918681 546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:07.920110 546 log.cc:824] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e: Log is configured to *not* fsync() on all Append() calls
Aug 09 22:37:07 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:07.923578 546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e: No bootstrap required, opened a new log
I20260809 22:37:07.929708 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:07.905605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.28.243.126:38467)
0809 22:37:07.906517 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:07.906529 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:07.906575 (+ 46us) server_negotiation.cc:407] Connection header received
0809 22:37:07.908303 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:07.908326 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:07.908826 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:07.909168 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:07.911366 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:07.913072 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:07.915054 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:07.915782 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:07.915927 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:07.921565 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:07.921573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:07.921589 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:07.921635 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:07.925102 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:07.925991 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:07.925998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:07.926005 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:07.926194 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:07.926637 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:07.926644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:07.926651 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:07.927223 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:07.927592 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:07.928047 (+ 455us) server_negotiation.cc:299] Negotiation successful
0809 22:37:07.928534 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":449,"threads_started":1}
I20260809 22:37:07.932909 546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb5bbe37f44e789cc15891fd47820e" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38467 } }
I20260809 22:37:07.933426 546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:07.933575 546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdb5bbe37f44e789cc15891fd47820e, State: Initialized, Role: FOLLOWER
I20260809 22:37:07.934217 546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb5bbe37f44e789cc15891fd47820e" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38467 } }
I20260809 22:37:07.934470 546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:07.934633 546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:07.934806 546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:07.937484 546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb5bbe37f44e789cc15891fd47820e" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38467 } }
I20260809 22:37:07.938009 546 leader_election.cc:302] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cdb5bbe37f44e789cc15891fd47820e; no voters:
I20260809 22:37:07.938697 546 leader_election.cc:288] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:07.938915 552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:07.939736 552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [term 1 LEADER]: Becoming Leader. State: Replica: 3cdb5bbe37f44e789cc15891fd47820e, State: Running, Role: LEADER
I20260809 22:37:07.940429 552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb5bbe37f44e789cc15891fd47820e" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38467 } }
I20260809 22:37:07.941067 546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:07.943596 554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cdb5bbe37f44e789cc15891fd47820e. Latest consensus state: current_term: 1 leader_uuid: "3cdb5bbe37f44e789cc15891fd47820e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb5bbe37f44e789cc15891fd47820e" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38467 } } }
I20260809 22:37:07.944550 554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:07.944579 553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cdb5bbe37f44e789cc15891fd47820e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdb5bbe37f44e789cc15891fd47820e" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38467 } } }
I20260809 22:37:07.944964 553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:07.946671 562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:07.957535 562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:07.967345 562 catalog_manager.cc:1347] Generated new cluster ID: 4aa9c4d922ca465a8b4664802426e7af
I20260809 22:37:07.967510 562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:07.981758 562 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:07.983171 562 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:07.993757 562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cdb5bbe37f44e789cc15891fd47820e: Generated new TSK 0
I20260809 22:37:07.994827 562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:37:08.060819 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38467
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:08.418537 576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:08.418851 576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:08.418952 576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:08.428153 576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:08.428292 576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:08.428412 576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:08.439563 576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:08.441071 576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:08.442656 576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:08.457190 582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:08.457389 581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:08.457368 584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:08.458236 576 server_base.cc:1034] running on GCE node
I20260809 22:37:08.458930 576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:08.460054 576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:08.461318 576 hybrid_clock.cc:648] HybridClock initialized: now 1786315028461269 us; error 36 us; skew 500 ppm
I20260809 22:37:08.461704 576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:08 dist-test-slave-rgb1 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:08.465258 576 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:37:08.466892 576 webserver.cc:460] Webserver started at http://127.28.243.65:45533/ using document root <none> and password file <none>
I20260809 22:37:08.467640 576 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:08.467803 576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:08.468138 576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:08.471349 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "7d97820dc46f45339127bfc56f988d9b"
format_stamp: "Formatted at 2026-08-09 22:37:08 on dist-test-slave-rgb1"
server_key: "a323051fd3b16d17feda01683f7b4a8b"
server_key_iv: "f01488e8a83194ee4b6df68c3ab12c63"
server_key_version: "encryptionkey@0"
I20260809 22:37:08.472643 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "7d97820dc46f45339127bfc56f988d9b"
format_stamp: "Formatted at 2026-08-09 22:37:08 on dist-test-slave-rgb1"
server_key: "a323051fd3b16d17feda01683f7b4a8b"
server_key_iv: "f01488e8a83194ee4b6df68c3ab12c63"
server_key_version: "encryptionkey@0"
I20260809 22:37:08.479199 576 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260809 22:37:08.483130 591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:08.484814 576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:08.485034 576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "7d97820dc46f45339127bfc56f988d9b"
format_stamp: "Formatted at 2026-08-09 22:37:08 on dist-test-slave-rgb1"
server_key: "a323051fd3b16d17feda01683f7b4a8b"
server_key_iv: "f01488e8a83194ee4b6df68c3ab12c63"
server_key_version: "encryptionkey@0"
I20260809 22:37:08.485273 576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:08.510547 576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:08.515250 576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:08.515583 576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:08.516701 576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:08.518198 576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:08.518308 576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:08.518482 576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:08.518559 576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:08.555320 576 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:46373
I20260809 22:37:08.555413 715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:46373 every 8 connection(s)
I20260809 22:37:08.556735 576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:08.567173 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 576
I20260809 22:37:08.567556 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:08.568468 29645 external_mini_cluster.cc:1428] Setting key 89092f35f99b473dd4f02b42155160a1
Aug 09 22:37:08 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315028, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:08.580034 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:08.559542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:46561 (local address 127.28.243.126:38467)
0809 22:37:08.560029 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:08.560036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:08.561141 (+ 1105us) server_negotiation.cc:407] Connection header received
0809 22:37:08.562989 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:08.562994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:08.563067 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:08.563171 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:08.564823 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:08.565626 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:08.567371 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:08.568076 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:08.568233 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:08.572289 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:08.572298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:08.572306 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:08.572355 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:08.576192 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:08.577190 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:08.577198 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:08.577204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:08.577360 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:08.577810 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:08.577818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:08.577824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:08.578217 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:08.578372 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:08.579308 (+ 936us) server_negotiation.cc:299] Negotiation successful
0809 22:37:08.579576 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260809 22:37:08.581187 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:08.560139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38467 (local address 127.28.243.65:46561)
0809 22:37:08.560918 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:08.561018 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:08.562714 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:08.563374 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:08.563383 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:08.563909 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:08.564610 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:08.564651 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:08.565790 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:08.565794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:08.567119 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:08.567132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:08.568346 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:08.568361 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:08.568673 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:08.569454 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:08.569478 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:08.572019 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:08.576362 (+ 4343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:08.576370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:08.576384 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:08.576950 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:08.577478 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:08.577482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:08.577486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:08.577686 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:08.578388 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:08.578394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:08.578766 (+ 372us) client_negotiation.cc:769] Sending connection context
0809 22:37:08.579131 (+ 365us) client_negotiation.cc:240] Negotiation successful
0809 22:37:08.579565 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":284,"threads_started":1}
I20260809 22:37:08.583048 716 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38467
I20260809 22:37:08.583418 716 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:08.584218 716 heartbeater.cc:511] Master 127.28.243.126:38467 requested a full tablet report, sending...
I20260809 22:37:08.586982 509 ts_manager.cc:194] Registered new tserver with Master: 7d97820dc46f45339127bfc56f988d9b (127.28.243.65:46373)
I20260809 22:37:08.589478 509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:46561
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:37:08.628772 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38467
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:08.973538 722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:08.973928 722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:08.974072 722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:08.983425 722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:08.983566 722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:08.983698 722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:08.995272 722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:08.996691 722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:08.998126 722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:09.013214 728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:09.013124 729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:09.013170 731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:09.014170 722 server_base.cc:1034] running on GCE node
I20260809 22:37:09.014860 722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:09.015997 722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:09.017248 722 hybrid_clock.cc:648] HybridClock initialized: now 1786315029017179 us; error 66 us; skew 500 ppm
I20260809 22:37:09.017622 722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:09.021174 722 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:37:09.022832 722 webserver.cc:460] Webserver started at http://127.28.243.66:43167/ using document root <none> and password file <none>
I20260809 22:37:09.023577 722 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:09.023717 722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:09.024061 722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:09.027484 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "d75f8621218e4fd2b8f353d17865feb1"
format_stamp: "Formatted at 2026-08-09 22:37:09 on dist-test-slave-rgb1"
server_key: "dc02c6f0619f741df0ab318d4f497e1b"
server_key_iv: "60aaa81a55b81c863da8caad8289b258"
server_key_version: "encryptionkey@0"
I20260809 22:37:09.028326 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "d75f8621218e4fd2b8f353d17865feb1"
format_stamp: "Formatted at 2026-08-09 22:37:09 on dist-test-slave-rgb1"
server_key: "dc02c6f0619f741df0ab318d4f497e1b"
server_key_iv: "60aaa81a55b81c863da8caad8289b258"
server_key_version: "encryptionkey@0"
I20260809 22:37:09.034173 722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 22:37:09.038990 738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:09.040825 722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:09.041049 722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "d75f8621218e4fd2b8f353d17865feb1"
format_stamp: "Formatted at 2026-08-09 22:37:09 on dist-test-slave-rgb1"
server_key: "dc02c6f0619f741df0ab318d4f497e1b"
server_key_iv: "60aaa81a55b81c863da8caad8289b258"
server_key_version: "encryptionkey@0"
I20260809 22:37:09.041266 722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:09.068302 722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:09.073450 722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:09.073804 722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:09.075026 722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:09.076653 722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:09.076732 722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:09.076879 722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:09.076933 722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:09.116526 722 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:40427
I20260809 22:37:09.116623 861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:40427 every 8 connection(s)
I20260809 22:37:09.118036 722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:09.124691 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 722
I20260809 22:37:09.124889 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:09.125489 29645 external_mini_cluster.cc:1428] Setting key f628ecda4bb55e37da811ba765635431
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:09.144071 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.120798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:57073 (local address 127.28.243.126:38467)
0809 22:37:09.121342 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:09.121347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:09.122392 (+ 1045us) server_negotiation.cc:407] Connection header received
0809 22:37:09.124357 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:09.124361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:09.124437 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:09.124556 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:09.127269 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.128148 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.130010 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.130957 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.131147 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.135490 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:09.135499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:09.135507 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:09.135616 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:09.139559 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.140983 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.140991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.140995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.141171 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.141898 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.141909 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.141911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.142299 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:09.142477 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:09.143540 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0809 22:37:09.143766 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":326,"threads_started":1}
I20260809 22:37:09.145920 866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.121467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38467 (local address 127.28.243.66:57073)
0809 22:37:09.122216 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:09.122265 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:09.124008 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:09.124871 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:09.124885 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:09.125822 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:09.126703 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:09.126724 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.128358 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.128364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:09.129828 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:09.129842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.131253 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.131259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:09.131500 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.132299 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:09.132325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:09.135045 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:09.139824 (+ 4779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:09.139839 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:09.139863 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:09.140693 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:09.141317 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:09.141325 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:09.141332 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:09.141661 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:09.142466 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:09.142478 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:09.143009 (+ 531us) client_negotiation.cc:769] Sending connection context
0809 22:37:09.143365 (+ 356us) client_negotiation.cc:240] Negotiation successful
0809 22:37:09.144072 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":280,"threads_started":1}
I20260809 22:37:09.148245 862 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38467
I20260809 22:37:09.148927 862 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:09.149902 862 heartbeater.cc:511] Master 127.28.243.126:38467 requested a full tablet report, sending...
I20260809 22:37:09.152037 509 ts_manager.cc:194] Registered new tserver with Master: d75f8621218e4fd2b8f353d17865feb1 (127.28.243.66:40427)
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
I20260809 22:37:09.153435 509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:57073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:37:09.189481 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38467
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:09.546597 869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:09.546988 869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:09.547188 869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:09.555054 869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:09.555222 869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:09.555398 869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:09.567230 869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:09.568854 869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:09.570416 869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:09.585363 874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:09.585446 877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:09.585530 875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:09.585810 869 server_base.cc:1034] running on GCE node
I20260809 22:37:09.586892 869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:09.587911 869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:09.589128 869 hybrid_clock.cc:648] HybridClock initialized: now 1786315029589082 us; error 30 us; skew 500 ppm
I20260809 22:37:09.589499 869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:09.593426 869 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:37:09.594017 716 heartbeater.cc:503] Master 127.28.243.126:38467 was elected leader, sending a full tablet report...
I20260809 22:37:09.595263 869 webserver.cc:460] Webserver started at http://127.28.243.67:38025/ using document root <none> and password file <none>
I20260809 22:37:09.596068 869 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:09.596226 869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:09.596508 869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:09.599624 869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "c678898dea7746e199d588e56cb4dea9"
format_stamp: "Formatted at 2026-08-09 22:37:09 on dist-test-slave-rgb1"
server_key: "9fe644be82967b4eb819e76ea554239b"
server_key_iv: "939857ac897f7d17f7bfa40cbac0316e"
server_key_version: "encryptionkey@0"
I20260809 22:37:09.600415 869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "c678898dea7746e199d588e56cb4dea9"
format_stamp: "Formatted at 2026-08-09 22:37:09 on dist-test-slave-rgb1"
server_key: "9fe644be82967b4eb819e76ea554239b"
server_key_iv: "939857ac897f7d17f7bfa40cbac0316e"
server_key_version: "encryptionkey@0"
I20260809 22:37:09.606704 869 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260809 22:37:09.611465 885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:09.613364 869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260809 22:37:09.613591 869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "c678898dea7746e199d588e56cb4dea9"
format_stamp: "Formatted at 2026-08-09 22:37:09 on dist-test-slave-rgb1"
server_key: "9fe644be82967b4eb819e76ea554239b"
server_key_iv: "939857ac897f7d17f7bfa40cbac0316e"
server_key_version: "encryptionkey@0"
I20260809 22:37:09.613821 869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:09.633927 869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:09.638718 869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:09.639065 869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:09.640136 869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:09.641811 869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:09.641891 869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:09.642079 869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:09.642153 869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:09.679409 869 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:40033
I20260809 22:37:09.679432 1007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:40033 every 8 connection(s)
I20260809 22:37:09.680804 869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:09.684978 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 869
I20260809 22:37:09.685205 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:09.685745 29645 external_mini_cluster.cc:1428] Setting key b5cc6e94a8bc51649233cd448f7e09b1
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:09.704779 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.683580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:45943 (local address 127.28.243.126:38467)
0809 22:37:09.684171 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:09.684187 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:09.685396 (+ 1209us) server_negotiation.cc:407] Connection header received
0809 22:37:09.687499 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:09.687506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:09.687616 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:09.687750 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:09.689675 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.690654 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.692490 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.692942 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.693049 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.697060 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:09.697067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:09.697070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:09.697112 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:09.700330 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.701779 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.701785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.701787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.701949 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.702628 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.702634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.702640 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.703026 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:09.703207 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:09.704073 (+ 866us) server_negotiation.cc:299] Negotiation successful
0809 22:37:09.704315 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":320,"threads_started":1}
I20260809 22:37:09.705719 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.684268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38467 (local address 127.28.243.67:45943)
0809 22:37:09.685145 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:09.685206 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:09.687221 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:09.688075 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:09.688084 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:09.688696 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:09.689459 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:09.689479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.691040 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.691044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:09.692335 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:09.692347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.693196 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.693202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:09.693486 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.694181 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:09.694206 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:09.696777 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:09.700535 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:09.700546 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:09.700587 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:09.701535 (+ 948us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:09.702137 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:09.702145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:09.702151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:09.702446 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:09.703174 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:09.703183 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:09.703676 (+ 493us) client_negotiation.cc:769] Sending connection context
0809 22:37:09.703975 (+ 299us) client_negotiation.cc:240] Negotiation successful
0809 22:37:09.704530 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":393,"threads_started":1}
I20260809 22:37:09.707242 1008 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38467
I20260809 22:37:09.707617 1008 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:09.708408 1008 heartbeater.cc:511] Master 127.28.243.126:38467 requested a full tablet report, sending...
I20260809 22:37:09.710014 509 ts_manager.cc:194] Registered new tserver with Master: c678898dea7746e199d588e56cb4dea9 (127.28.243.67:40033)
I20260809 22:37:09.710877 509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:45943
I20260809 22:37:09.716135 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:09.776223 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.759835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.28.243.126:38467)
0809 22:37:09.760116 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:09.760121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:09.760515 (+ 394us) server_negotiation.cc:407] Connection header received
0809 22:37:09.760719 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:09.760724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:09.760852 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:09.760949 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:09.762658 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.763401 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.764947 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.765467 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.765585 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.770165 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:09.770170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:09.770174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:09.770213 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:09.773296 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.774112 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.774124 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.774126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.774256 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.774690 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.774699 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.774701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.775064 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:09.775335 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:09.775664 (+ 329us) server_negotiation.cc:299] Negotiation successful
0809 22:37:09.775902 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
W20260809 22:37:09.776924 509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33604
I20260809 22:37:09.779066 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 576
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:37:09.797381 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.780380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.28.243.65:46373)
0809 22:37:09.781182 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:09.781190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:09.781217 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:37:09.781282 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:09.781293 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:09.781518 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:09.781675 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:09.783080 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.784197 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.785717 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.786387 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.786500 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.791152 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:09.791164 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:09.791178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:09.791217 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:09.794529 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.795320 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.795329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.795331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.795463 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.795854 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.795857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.795859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.796220 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:09.796465 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:09.796843 (+ 378us) server_negotiation.cc:299] Negotiation successful
0809 22:37:09.797059 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":408,"threads_started":1}
W20260809 22:37:09.798853 711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57846
W20260809 22:37:09.800082 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:37:09.800329 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 576
I20260809 22:37:09.818638 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 722
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:37:09.837972 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.819721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.28.243.66:40427)
0809 22:37:09.820296 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:09.820303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:09.820328 (+ 25us) server_negotiation.cc:407] Connection header received
0809 22:37:09.820390 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:09.820396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:09.820751 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:09.820927 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:09.822208 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.823226 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.824551 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.825206 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.825355 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.830557 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:09.830572 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:09.830587 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:09.830642 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:09.834429 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.835275 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.835283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.835286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.835447 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.835975 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.835979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.835981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.836375 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:09.836703 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:09.837157 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 22:37:09.837489 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":268,"threads_started":1}
W20260809 22:37:09.839437 854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52434
W20260809 22:37:09.840739 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:37:09.840891 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 722
I20260809 22:37:09.859201 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 869
Aug 09 22:37:09 dist-test-slave-rgb1 krb5kdc[460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:37:09.878574 1025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:09.860395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.28.243.67:40033)
0809 22:37:09.861230 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:09.861237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:09.861269 (+ 32us) server_negotiation.cc:407] Connection header received
0809 22:37:09.861335 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:09.861342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:09.861552 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:09.861703 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:09.863223 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.864367 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.865751 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:09.866331 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:09.866432 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:09.871434 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:09.871447 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:09.871463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:09.871502 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:09.875141 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.875964 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.875972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.875975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.876113 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:09.876711 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:09.876721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:09.876727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:09.877199 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:09.877478 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:09.877875 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 22:37:09.878216 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":412,"threads_started":1}
W20260809 22:37:09.880016 1003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41146
W20260809 22:37:09.881376 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:37:09.881551 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 869
I20260809 22:37:09.899837 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 473
I20260809 22:37:09.901192 540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:10.010778 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 473
2026-08-09T22:37:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4036 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:10 dist-test-slave-rgb1 krb5kdc[1029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:10 dist-test-slave-rgb1 krb5kdc[1029](info): set up 2 sockets
Aug 09 22:37:10 dist-test-slave-rgb1 krb5kdc[1029](info): commencing operation
krb5kdc: starting...
W20260809 22:37:11.203233 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:11 dist-test-slave-rgb1 krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:11.469815 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:45657
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:45657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:11.823132 1042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:11.823479 1042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:11.823589 1042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:11.831467 1042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:11.831646 1042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:11.831718 1042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:11.831779 1042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:11.831826 1042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:11.842604 1042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:45657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:45657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:11.844062 1042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:11.845475 1042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:11.858431 1050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:11.859030 1047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:11.859089 1048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:11.859579 1042 server_base.cc:1034] running on GCE node
I20260809 22:37:11.860435 1042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:11.862097 1042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:11.863344 1042 hybrid_clock.cc:648] HybridClock initialized: now 1786315031863293 us; error 42 us; skew 500 ppm
I20260809 22:37:11.863758 1042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:11 dist-test-slave-rgb1 krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315031, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:11.871320 1042 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:11.873409 1042 webserver.cc:460] Webserver started at http://127.28.243.126:44815/ using document root <none> and password file <none>
I20260809 22:37:11.874228 1042 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:11.874399 1042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:11.874761 1042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:11.878216 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "8a020bf55ffa4eedaf492147543f4965"
format_stamp: "Formatted at 2026-08-09 22:37:11 on dist-test-slave-rgb1"
server_key: "2677297840aeed2781b19efccf4dc281"
server_key_iv: "61dfb541d53ec5c7c065384346e555bc"
server_key_version: "encryptionkey@0"
I20260809 22:37:11.879125 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "8a020bf55ffa4eedaf492147543f4965"
format_stamp: "Formatted at 2026-08-09 22:37:11 on dist-test-slave-rgb1"
server_key: "2677297840aeed2781b19efccf4dc281"
server_key_iv: "61dfb541d53ec5c7c065384346e555bc"
server_key_version: "encryptionkey@0"
I20260809 22:37:11.885015 1042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 22:37:11.888681 1058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:11.890264 1042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:37:11.890475 1042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "8a020bf55ffa4eedaf492147543f4965"
format_stamp: "Formatted at 2026-08-09 22:37:11 on dist-test-slave-rgb1"
server_key: "2677297840aeed2781b19efccf4dc281"
server_key_iv: "61dfb541d53ec5c7c065384346e555bc"
server_key_version: "encryptionkey@0"
I20260809 22:37:11.890693 1042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:11.903523 1042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:11.908841 1042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:11.909214 1042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:11.930485 1042 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:45657
I20260809 22:37:11.930532 1112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:45657 every 8 connection(s)
I20260809 22:37:11.931995 1042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:11.935165 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 1042
I20260809 22:37:11.935447 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:11.936381 29645 external_mini_cluster.cc:1428] Setting key 0c5d03526a84c70dab9bb4d6e567e8ab
I20260809 22:37:11.937750 1113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 22:37:11 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:11.952401 1113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965: Bootstrap starting.
I20260809 22:37:11.957381 1113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:11.958864 1113 log.cc:824] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:11.962925 1113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965: No bootstrap required, opened a new log
I20260809 22:37:11.963263 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:11.938745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.28.243.126:45657)
0809 22:37:11.939684 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:11.939699 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:11.939756 (+ 57us) server_negotiation.cc:407] Connection header received
0809 22:37:11.941989 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:11.942013 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:11.942450 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:11.942917 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:11.944453 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:11.945949 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:11.947232 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:11.947933 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:11.948154 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:11.953061 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:11.953073 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:11.953096 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:11.953149 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:11.957770 (+ 4621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:11.958662 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:11.958672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:11.958682 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:11.958947 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:11.959549 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:11.959558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:11.959564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:11.960517 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:11.960873 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:11.961304 (+ 431us) server_negotiation.cc:299] Negotiation successful
0809 22:37:11.961795 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":421,"threads_started":1}
I20260809 22:37:11.972684 1113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a020bf55ffa4eedaf492147543f4965" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45657 } }
I20260809 22:37:11.973119 1113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:11.973192 1113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a020bf55ffa4eedaf492147543f4965, State: Initialized, Role: FOLLOWER
I20260809 22:37:11.973760 1113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a020bf55ffa4eedaf492147543f4965" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45657 } }
I20260809 22:37:11.973976 1113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:11.974080 1113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:11.974223 1113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:11.977141 1113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a020bf55ffa4eedaf492147543f4965" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45657 } }
I20260809 22:37:11.977672 1113 leader_election.cc:302] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a020bf55ffa4eedaf492147543f4965; no voters:
I20260809 22:37:11.978296 1113 leader_election.cc:288] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:11.978456 1119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:11.979707 1119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [term 1 LEADER]: Becoming Leader. State: Replica: 8a020bf55ffa4eedaf492147543f4965, State: Running, Role: LEADER
I20260809 22:37:11.980330 1119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a020bf55ffa4eedaf492147543f4965" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45657 } }
I20260809 22:37:11.980898 1113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:11.983291 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a020bf55ffa4eedaf492147543f4965. Latest consensus state: current_term: 1 leader_uuid: "8a020bf55ffa4eedaf492147543f4965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a020bf55ffa4eedaf492147543f4965" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45657 } } }
I20260809 22:37:11.984148 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:11.983855 1121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a020bf55ffa4eedaf492147543f4965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a020bf55ffa4eedaf492147543f4965" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 45657 } } }
I20260809 22:37:11.984298 1121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:11.985172 1127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:11.995879 1127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:12.008942 1127 catalog_manager.cc:1347] Generated new cluster ID: 152a3a616f8d46179565ea4ff34f2687
I20260809 22:37:12.009111 1127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:12.026244 1127 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:12.027622 1127 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:12.035463 1127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a020bf55ffa4eedaf492147543f4965: Generated new TSK 0
I20260809 22:37:12.036850 1127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:37:12.106256 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45657
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:12.467406 1141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:12.467801 1141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:12.467971 1141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:12.476078 1141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:12.476296 1141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:12.476506 1141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:12.487663 1141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:12.489246 1141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:12.490689 1141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:12.511598 1149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:12.511642 1148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:12.511615 1152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:12.513124 1141 server_base.cc:1034] running on GCE node
I20260809 22:37:12.513872 1141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:12.515069 1141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:12.516410 1141 hybrid_clock.cc:648] HybridClock initialized: now 1786315032516292 us; error 53 us; skew 500 ppm
I20260809 22:37:12.516937 1141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:12 dist-test-slave-rgb1 krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315032, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:12.521147 1141 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:37:12.523147 1141 webserver.cc:460] Webserver started at http://127.28.243.65:39301/ using document root <none> and password file <none>
I20260809 22:37:12.523965 1141 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:12.524137 1141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:12.524533 1141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:12.528126 1141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "7094e12ea1d2439db90f3525ebb2fd3f"
format_stamp: "Formatted at 2026-08-09 22:37:12 on dist-test-slave-rgb1"
server_key: "4f3e8653b04b25137496155e0ad6982e"
server_key_iv: "9323b16b12c1f0020eea923f15e51ce2"
server_key_version: "encryptionkey@0"
I20260809 22:37:12.529088 1141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "7094e12ea1d2439db90f3525ebb2fd3f"
format_stamp: "Formatted at 2026-08-09 22:37:12 on dist-test-slave-rgb1"
server_key: "4f3e8653b04b25137496155e0ad6982e"
server_key_iv: "9323b16b12c1f0020eea923f15e51ce2"
server_key_version: "encryptionkey@0"
I20260809 22:37:12.535185 1141 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260809 22:37:12.539369 1161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:12.541050 1141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:12.541244 1141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "7094e12ea1d2439db90f3525ebb2fd3f"
format_stamp: "Formatted at 2026-08-09 22:37:12 on dist-test-slave-rgb1"
server_key: "4f3e8653b04b25137496155e0ad6982e"
server_key_iv: "9323b16b12c1f0020eea923f15e51ce2"
server_key_version: "encryptionkey@0"
I20260809 22:37:12.541474 1141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:12.553767 1141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:12.558636 1141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:12.558959 1141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:12.559978 1141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:12.561513 1141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:12.561647 1141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:12.561806 1141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:12.561882 1141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:12.600732 1141 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:33541
I20260809 22:37:12.600759 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:33541 every 8 connection(s)
I20260809 22:37:12.602124 1141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:12.603732 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 1141
I20260809 22:37:12.603976 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:12.604892 29645 external_mini_cluster.cc:1428] Setting key 6514ac799a610f395ebc3f7420fcb204
Aug 09 22:37:12 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315032, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:12.629544 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:12.605198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:60791 (local address 127.28.243.126:45657)
0809 22:37:12.606234 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:12.606244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:12.609468 (+ 3224us) server_negotiation.cc:407] Connection header received
0809 22:37:12.611501 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:12.611507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:12.611602 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:12.611738 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:12.613426 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:12.614179 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:12.615773 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:12.616509 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:12.616700 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:12.621240 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:12.621249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:12.621257 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:12.621314 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:12.625400 (+ 4086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:12.626398 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:12.626406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:12.626413 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:12.626598 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:12.627303 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:12.627311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:12.627316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:12.627726 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:12.627907 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:12.628798 (+ 891us) server_negotiation.cc:299] Negotiation successful
0809 22:37:12.629077 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":709,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:37:12.630393 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:12.605696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45657 (local address 127.28.243.65:60791)
0809 22:37:12.609251 (+ 3555us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:12.609313 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:12.611190 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:12.611997 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:12.612006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:12.612621 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:12.613242 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:12.613260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:12.614348 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:12.614353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:12.615611 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:12.615629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:12.616841 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:12.616847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:12.617086 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:12.617875 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:12.617900 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:12.620968 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:12.625637 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:12.625655 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:12.625671 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:12.626220 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:12.626826 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:12.626894 (+ 68us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:12.626896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:12.627149 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:12.627945 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:12.627952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:12.628325 (+ 373us) client_negotiation.cc:769] Sending connection context
0809 22:37:12.628663 (+ 338us) client_negotiation.cc:240] Negotiation successful
0809 22:37:12.629119 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3253,"thread_start_us":3005,"threads_started":1}
I20260809 22:37:12.632244 1284 heartbeater.cc:347] Connected to a master server at 127.28.243.126:45657
I20260809 22:37:12.632740 1284 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
I20260809 22:37:12.633723 1284 heartbeater.cc:511] Master 127.28.243.126:45657 requested a full tablet report, sending...
I20260809 22:37:12.636313 1076 ts_manager.cc:194] Registered new tserver with Master: 7094e12ea1d2439db90f3525ebb2fd3f (127.28.243.65:33541)
I20260809 22:37:12.638233 1076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:60791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:37:12.664904 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45657
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:13.023671 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:13.024035 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:13.024200 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:13.031853 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:13.032018 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:13.032210 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:13.042850 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:13.044355 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:13.045876 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:13.062216 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:13.062264 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:13.062244 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:13.063882 1290 server_base.cc:1034] running on GCE node
I20260809 22:37:13.064761 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:13.066056 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:13.067337 1290 hybrid_clock.cc:648] HybridClock initialized: now 1786315033067285 us; error 48 us; skew 500 ppm
I20260809 22:37:13.067782 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:13.071736 1290 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:37:13.073724 1290 webserver.cc:460] Webserver started at http://127.28.243.66:39777/ using document root <none> and password file <none>
I20260809 22:37:13.074589 1290 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:13.074764 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:13.075161 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:13.078603 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "30fc5296b0d94c149cd233272f403cf2"
format_stamp: "Formatted at 2026-08-09 22:37:13 on dist-test-slave-rgb1"
server_key: "7dde84b4f92e9f57406741c237e14d9d"
server_key_iv: "b3f5c695b8c227fc274aef17a375f793"
server_key_version: "encryptionkey@0"
I20260809 22:37:13.079612 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "30fc5296b0d94c149cd233272f403cf2"
format_stamp: "Formatted at 2026-08-09 22:37:13 on dist-test-slave-rgb1"
server_key: "7dde84b4f92e9f57406741c237e14d9d"
server_key_iv: "b3f5c695b8c227fc274aef17a375f793"
server_key_version: "encryptionkey@0"
I20260809 22:37:13.085979 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 22:37:13.090701 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:13.092485 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:37:13.092751 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "30fc5296b0d94c149cd233272f403cf2"
format_stamp: "Formatted at 2026-08-09 22:37:13 on dist-test-slave-rgb1"
server_key: "7dde84b4f92e9f57406741c237e14d9d"
server_key_iv: "b3f5c695b8c227fc274aef17a375f793"
server_key_version: "encryptionkey@0"
I20260809 22:37:13.093022 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:13.107563 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:13.113284 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:13.113683 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:13.114838 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:13.116816 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:13.116950 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:13.117120 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:13.117202 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:13.159641 1290 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:41483
I20260809 22:37:13.159696 1430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:41483 every 8 connection(s)
I20260809 22:37:13.161384 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:13.170537 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 1290
I20260809 22:37:13.170811 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:13.171411 29645 external_mini_cluster.cc:1428] Setting key 57f4ae9ed304b57d6a4d6be81dcb67b7
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:13.186527 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.164544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:53701 (local address 127.28.243.126:45657)
0809 22:37:13.165095 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:13.165100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:13.165885 (+ 785us) server_negotiation.cc:407] Connection header received
0809 22:37:13.167849 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:13.167855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:13.167949 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:13.168084 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:13.169761 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.170525 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.172378 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.173083 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.173239 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.177774 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:13.177784 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:13.177791 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:13.177842 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:13.181962 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.183452 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.183457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.183459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.183661 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.184292 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.184296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.184298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.184737 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:13.184870 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:13.185958 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0809 22:37:13.186189 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":257,"threads_started":1}
I20260809 22:37:13.188189 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.165005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45657 (local address 127.28.243.66:53701)
0809 22:37:13.165712 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:13.165752 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:13.167559 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:13.168341 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:13.168350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:13.168969 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:13.169567 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:13.169591 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.170722 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.170730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:13.172165 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:13.172210 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.173368 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.173377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:13.173690 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.174446 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:13.174480 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:13.177458 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:13.182225 (+ 4767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:13.182240 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:13.182267 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:13.183233 (+ 966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:13.183821 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:13.183829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:13.183836 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:13.184142 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:13.184880 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:13.184893 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:13.185444 (+ 551us) client_negotiation.cc:769] Sending connection context
0809 22:37:13.185787 (+ 343us) client_negotiation.cc:240] Negotiation successful
0809 22:37:13.186466 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":288,"threads_started":1}
I20260809 22:37:13.190130 1431 heartbeater.cc:347] Connected to a master server at 127.28.243.126:45657
I20260809 22:37:13.190588 1431 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:13.191452 1431 heartbeater.cc:511] Master 127.28.243.126:45657 requested a full tablet report, sending...
I20260809 22:37:13.193379 1076 ts_manager.cc:194] Registered new tserver with Master: 30fc5296b0d94c149cd233272f403cf2 (127.28.243.66:41483)
I20260809 22:37:13.194917 1076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:53701
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:37:13.234021 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45657
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:13.595659 1438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:13.595973 1438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:13.596074 1438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:13.605037 1438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:13.605223 1438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:13.605405 1438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:13.615877 1438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:45657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:13.617350 1438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:13.618736 1438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:13.632892 1443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:13.632925 1444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:13.632854 1446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:13.634076 1438 server_base.cc:1034] running on GCE node
I20260809 22:37:13.634755 1438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:13.635933 1438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:13.637180 1438 hybrid_clock.cc:648] HybridClock initialized: now 1786315033637124 us; error 50 us; skew 500 ppm
I20260809 22:37:13.637570 1438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:13.641338 1438 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:37:13.642426 1284 heartbeater.cc:503] Master 127.28.243.126:45657 was elected leader, sending a full tablet report...
I20260809 22:37:13.643461 1438 webserver.cc:460] Webserver started at http://127.28.243.67:40359/ using document root <none> and password file <none>
I20260809 22:37:13.644273 1438 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:13.644456 1438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:13.644866 1438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:13.648056 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "cfe45b114c594533a8b624aab92c031e"
format_stamp: "Formatted at 2026-08-09 22:37:13 on dist-test-slave-rgb1"
server_key: "b7f7fd720feaab2e02737743c9513cc6"
server_key_iv: "040ced360d6da8b9bc75578066cb31cc"
server_key_version: "encryptionkey@0"
I20260809 22:37:13.648960 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "cfe45b114c594533a8b624aab92c031e"
format_stamp: "Formatted at 2026-08-09 22:37:13 on dist-test-slave-rgb1"
server_key: "b7f7fd720feaab2e02737743c9513cc6"
server_key_iv: "040ced360d6da8b9bc75578066cb31cc"
server_key_version: "encryptionkey@0"
I20260809 22:37:13.654618 1438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 22:37:13.658999 1455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:13.660549 1438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 22:37:13.660794 1438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "cfe45b114c594533a8b624aab92c031e"
format_stamp: "Formatted at 2026-08-09 22:37:13 on dist-test-slave-rgb1"
server_key: "b7f7fd720feaab2e02737743c9513cc6"
server_key_iv: "040ced360d6da8b9bc75578066cb31cc"
server_key_version: "encryptionkey@0"
I20260809 22:37:13.661085 1438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:13.687108 1438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:13.692200 1438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:13.692487 1438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:13.693557 1438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:13.695111 1438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:13.695279 1438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:13.695390 1438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:13.695437 1438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:13.732978 1438 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:40867
I20260809 22:37:13.733024 1578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:40867 every 8 connection(s)
I20260809 22:37:13.734356 1438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:13.740119 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 1438
I20260809 22:37:13.740331 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:13.740948 29645 external_mini_cluster.cc:1428] Setting key 9dddd75825c0810428595d69e37b16ec
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:13.758937 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.737376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:39595 (local address 127.28.243.126:45657)
0809 22:37:13.737921 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:13.737926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:13.738984 (+ 1058us) server_negotiation.cc:407] Connection header received
0809 22:37:13.741240 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:13.741247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:13.741374 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:13.741502 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:13.743602 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.744662 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.746530 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.746965 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.747105 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.750929 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:13.750934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:13.750938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:13.750972 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:13.753817 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.754878 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.754885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.754890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.755136 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.755892 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.755899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.755902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.756371 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:13.756711 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:13.757701 (+ 990us) server_negotiation.cc:299] Negotiation successful
0809 22:37:13.757985 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":317,"threads_started":1}
I20260809 22:37:13.760030 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.737910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:45657 (local address 127.28.243.67:39595)
0809 22:37:13.738777 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:13.738829 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:13.740881 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:13.741828 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:13.741847 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:13.742621 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:13.743390 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:13.743411 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.744873 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.744877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:13.746365 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:13.746378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.747398 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.747404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:13.747621 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.748373 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:13.748399 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:13.750710 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:13.753973 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:13.753981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:13.753995 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:13.754527 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:13.755350 (+ 823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:13.755355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:13.755358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:13.755662 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:13.756658 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:13.756677 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:13.757195 (+ 518us) client_negotiation.cc:769] Sending connection context
0809 22:37:13.757585 (+ 390us) client_negotiation.cc:240] Negotiation successful
0809 22:37:13.758717 (+ 1132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":315,"threads_started":1}
I20260809 22:37:13.761598 1579 heartbeater.cc:347] Connected to a master server at 127.28.243.126:45657
I20260809 22:37:13.761965 1579 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:13.762799 1579 heartbeater.cc:511] Master 127.28.243.126:45657 requested a full tablet report, sending...
I20260809 22:37:13.764600 1076 ts_manager.cc:194] Registered new tserver with Master: cfe45b114c594533a8b624aab92c031e (127.28.243.67:40867)
I20260809 22:37:13.765540 1076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:39595
I20260809 22:37:13.771188 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:37:13.834249 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.816628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.28.243.65:33541)
0809 22:37:13.817351 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:13.817367 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:13.817396 (+ 29us) server_negotiation.cc:407] Connection header received
0809 22:37:13.817462 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:13.817468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:13.817690 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:13.817827 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:13.819487 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.820693 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.822122 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.822737 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.822853 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.827511 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:13.827528 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:13.827543 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:13.827596 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:13.830866 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.831709 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.831718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.831720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.831869 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.832420 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.832425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.832427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.832935 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:13.833206 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:13.833565 (+ 359us) server_negotiation.cc:299] Negotiation successful
0809 22:37:13.833791 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":404,"threads_started":1}
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:13.856920 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.839735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.28.243.126:45657)
0809 22:37:13.840023 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:13.840027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:13.840093 (+ 66us) server_negotiation.cc:407] Connection header received
0809 22:37:13.840220 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:13.840224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:13.840294 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:13.840379 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:13.841954 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.842765 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.844480 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.845251 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.845376 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.849967 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:13.849973 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:13.849976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:13.850020 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:13.853298 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.854177 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.854182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.854184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.854362 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.854977 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.854981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.854983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.855419 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:13.855651 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:13.855974 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 22:37:13.856509 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260809 22:37:13.857643 1076 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54936
I20260809 22:37:13.859153 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 1141
I20260809 22:37:13.875469 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.860369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.28.243.65:33541)
0809 22:37:13.860766 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:13.860775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:13.861058 (+ 283us) server_negotiation.cc:407] Connection header received
0809 22:37:13.861318 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:13.861324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:13.861399 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:13.861524 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:13.863420 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.864161 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.865603 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.866111 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.866261 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.869496 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:13.869502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:13.869505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:13.869547 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:13.872692 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.873476 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.873482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.873487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.873630 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:13.874037 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:13.874041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:13.874043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:13.874302 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:13.874556 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:13.874896 (+ 340us) server_negotiation.cc:299] Negotiation successful
0809 22:37:13.875143 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 22:37:13.876199 1278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:13.962448 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 1141
I20260809 22:37:13.982765 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 1290
Aug 09 22:37:13 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:37:14.004380 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:13.983836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.28.243.66:41483)
0809 22:37:13.984451 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:13.984458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:13.984484 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:37:13.984547 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:13.984625 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:13.984886 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:13.985021 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:13.986424 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.987529 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.989448 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:13.990036 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:13.990164 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:13.996516 (+ 6352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:13.996539 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:13.996587 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:13.996642 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:14.000371 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:14.001602 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:14.001611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:14.001613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:14.001784 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:14.002364 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:14.002368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:14.002371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:14.002823 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:14.003112 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:14.003556 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 22:37:14.003919 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":333,"threads_started":1}
I20260809 22:37:14.005925 1426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:14.093340 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 1290
I20260809 22:37:14.112238 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 1438
Aug 09 22:37:14 dist-test-slave-rgb1 krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:37:14.132546 1596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:14.113409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.28.243.67:40867)
0809 22:37:14.114166 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:14.114176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:14.114212 (+ 36us) server_negotiation.cc:407] Connection header received
0809 22:37:14.114279 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:14.114285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:14.114674 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:14.114922 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:14.116603 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:14.117866 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:14.119450 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:14.120150 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:14.120331 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:14.126037 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:14.126056 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:14.126071 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:14.126113 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:14.129403 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:14.130228 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:14.130237 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:14.130240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:14.130382 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:14.130779 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:14.130785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:14.130788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:14.131192 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:14.131546 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:14.131867 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 22:37:14.132178 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":388,"threads_started":1}
I20260809 22:37:14.134135 1574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:14.221735 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 1438
I20260809 22:37:14.240526 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 1042
I20260809 22:37:14.241987 1108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:14.349261 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 1042
2026-08-09T22:37:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4337 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:14 dist-test-slave-rgb1 krb5kdc[1602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:14 dist-test-slave-rgb1 krb5kdc[1602](info): set up 2 sockets
Aug 09 22:37:14 dist-test-slave-rgb1 krb5kdc[1602](info): commencing operation
krb5kdc: starting...
W20260809 22:37:15.575605 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:15 dist-test-slave-rgb1 krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:15.841480 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34135
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:16.216146 1616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:16.216496 1616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:16.216672 1616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:16.224179 1616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:16.224341 1616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:16.224401 1616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:16.224522 1616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:16.224611 1616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:16.234989 1616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:16.236466 1616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:16.237859 1616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:16.250610 1622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:16.250548 1621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:16.250644 1625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:16.252090 1616 server_base.cc:1034] running on GCE node
I20260809 22:37:16.253131 1616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:16.254663 1616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:16.255895 1616 hybrid_clock.cc:648] HybridClock initialized: now 1786315036255863 us; error 70 us; skew 500 ppm
I20260809 22:37:16.256345 1616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:16 dist-test-slave-rgb1 krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:16.260183 1616 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:16.261740 1616 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:16.261894 1616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:16.262271 1616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:16.265642 1616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "25bc1f6e07184d0582314f19d9d3526a"
format_stamp: "Formatted at 2026-08-09 22:37:16 on dist-test-slave-rgb1"
server_key: "19d5316ae4486493b720ae3fcdd42e09"
server_key_iv: "7e141278602c2302a191ce231f28eac8"
server_key_version: "encryptionkey@0"
I20260809 22:37:16.266556 1616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "25bc1f6e07184d0582314f19d9d3526a"
format_stamp: "Formatted at 2026-08-09 22:37:16 on dist-test-slave-rgb1"
server_key: "19d5316ae4486493b720ae3fcdd42e09"
server_key_iv: "7e141278602c2302a191ce231f28eac8"
server_key_version: "encryptionkey@0"
I20260809 22:37:16.273053 1616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:37:16.277824 1632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:16.279570 1616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:37:16.279811 1616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "25bc1f6e07184d0582314f19d9d3526a"
format_stamp: "Formatted at 2026-08-09 22:37:16 on dist-test-slave-rgb1"
server_key: "19d5316ae4486493b720ae3fcdd42e09"
server_key_iv: "7e141278602c2302a191ce231f28eac8"
server_key_version: "encryptionkey@0"
I20260809 22:37:16.280113 1616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:16.298043 1616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:16.303114 1616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:16.303452 1616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:16.324308 1616 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:34135
I20260809 22:37:16.324330 1687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:34135 every 8 connection(s)
I20260809 22:37:16.325414 1616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:16.329825 1688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:16.330340 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 1616
I20260809 22:37:16.330608 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:16.331489 29645 external_mini_cluster.cc:1428] Setting key 33ff1b40ce624eb99d0a8415e7fe0423
I20260809 22:37:16.346568 1688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a: Bootstrap starting.
Aug 09 22:37:16 dist-test-slave-rgb1 krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:16.350714 1688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:16.351899 1688 log.cc:824] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:16.355387 1688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a: No bootstrap required, opened a new log
I20260809 22:37:16.359598 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:16.334058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.28.243.126:34135)
0809 22:37:16.335039 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:16.335053 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:16.335096 (+ 43us) server_negotiation.cc:407] Connection header received
0809 22:37:16.337270 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:16.337298 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:16.337785 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:16.338194 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:16.340540 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:16.342326 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:16.344396 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:16.345197 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:16.345344 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:16.350524 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:16.350533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:16.350549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:16.350592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:16.354477 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:16.355321 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:16.355331 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:16.355342 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:16.355655 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:16.356133 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:16.356139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:16.356142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:16.356914 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:16.357208 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:16.357637 (+ 429us) server_negotiation.cc:299] Negotiation successful
0809 22:37:16.358106 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":482,"threads_started":1}
I20260809 22:37:16.365391 1688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25bc1f6e07184d0582314f19d9d3526a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34135 } }
I20260809 22:37:16.365774 1688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:16.365868 1688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bc1f6e07184d0582314f19d9d3526a, State: Initialized, Role: FOLLOWER
I20260809 22:37:16.366497 1688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25bc1f6e07184d0582314f19d9d3526a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34135 } }
I20260809 22:37:16.366737 1688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:16.366900 1688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:16.367112 1688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:16.369730 1688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25bc1f6e07184d0582314f19d9d3526a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34135 } }
I20260809 22:37:16.370235 1688 leader_election.cc:302] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25bc1f6e07184d0582314f19d9d3526a; no voters:
I20260809 22:37:16.370995 1688 leader_election.cc:288] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:16.371172 1693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:16.372159 1693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [term 1 LEADER]: Becoming Leader. State: Replica: 25bc1f6e07184d0582314f19d9d3526a, State: Running, Role: LEADER
I20260809 22:37:16.372861 1693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25bc1f6e07184d0582314f19d9d3526a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34135 } }
I20260809 22:37:16.373721 1688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:16.376225 1695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25bc1f6e07184d0582314f19d9d3526a. Latest consensus state: current_term: 1 leader_uuid: "25bc1f6e07184d0582314f19d9d3526a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25bc1f6e07184d0582314f19d9d3526a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34135 } } }
I20260809 22:37:16.376654 1695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:16.378473 1702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:16.378412 1694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25bc1f6e07184d0582314f19d9d3526a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25bc1f6e07184d0582314f19d9d3526a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34135 } } }
I20260809 22:37:16.379060 1694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:16.385223 1702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:16.395498 1702 catalog_manager.cc:1347] Generated new cluster ID: ba1bba890ca44dd8ac74e3455c48db0c
I20260809 22:37:16.395648 1702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:16.414673 1702 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:16.416054 1702 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:16.430076 1702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a: Generated new TSK 0
I20260809 22:37:16.431043 1702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:37:16.518592 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34135
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:16.902549 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:16.902945 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:16.903157 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:16.911451 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:16.911607 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:16.911866 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:16.922693 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:16.924198 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:16.925786 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:16.940423 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:16.940496 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:16.940470 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:16.941763 1716 server_base.cc:1034] running on GCE node
I20260809 22:37:16.942497 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:16.943490 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:16.944712 1716 hybrid_clock.cc:648] HybridClock initialized: now 1786315036944658 us; error 49 us; skew 500 ppm
I20260809 22:37:16.945164 1716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:16 dist-test-slave-rgb1 krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:16.954455 1716 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:37:16.955981 1716 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:16.956135 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:16.956485 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:16.959776 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "22c40d8c5f8549c9b76f7a614213a223"
format_stamp: "Formatted at 2026-08-09 22:37:16 on dist-test-slave-rgb1"
server_key: "2f9af5bea227e673b2311c62cd31029b"
server_key_iv: "dfb7047f78e34ee8148308f2fdaeced4"
server_key_version: "encryptionkey@0"
I20260809 22:37:16.960839 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "22c40d8c5f8549c9b76f7a614213a223"
format_stamp: "Formatted at 2026-08-09 22:37:16 on dist-test-slave-rgb1"
server_key: "2f9af5bea227e673b2311c62cd31029b"
server_key_iv: "dfb7047f78e34ee8148308f2fdaeced4"
server_key_version: "encryptionkey@0"
I20260809 22:37:16.966601 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:37:16.970599 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:16.972218 1716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:37:16.972437 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "22c40d8c5f8549c9b76f7a614213a223"
format_stamp: "Formatted at 2026-08-09 22:37:16 on dist-test-slave-rgb1"
server_key: "2f9af5bea227e673b2311c62cd31029b"
server_key_iv: "dfb7047f78e34ee8148308f2fdaeced4"
server_key_version: "encryptionkey@0"
I20260809 22:37:16.972748 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:16.986289 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:16.991041 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:16.991356 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:16.992450 1716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:16.994065 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:16.994230 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:16.994369 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:16.994463 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:17.035593 1716 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:41503
I20260809 22:37:17.035694 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:41503 every 8 connection(s)
I20260809 22:37:17.036651 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:17.039685 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 1716
I20260809 22:37:17.040102 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:17.041069 29645 external_mini_cluster.cc:1428] Setting key 05b0df94880dcc59981b3648e71b28b1
Aug 09 22:37:17 dist-test-slave-rgb1 krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315036, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:17.063373 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:17.039552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:51393 (local address 127.28.243.126:34135)
0809 22:37:17.040162 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:17.040171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:17.041477 (+ 1306us) server_negotiation.cc:407] Connection header received
0809 22:37:17.043826 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:17.043833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:17.043983 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:17.044125 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:17.046180 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.047329 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.049500 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.050191 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.050348 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:17.054449 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:17.054458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:17.054466 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:17.054524 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:17.058515 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:17.059735 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:17.059743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:17.059749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:17.059938 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:17.060447 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:17.060455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:17.060461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:17.061150 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:17.061378 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:17.062384 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0809 22:37:17.062734 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":301,"threads_started":1}
I20260809 22:37:17.064472 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:17.040185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34135 (local address 127.28.243.65:51393)
0809 22:37:17.041243 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:17.041312 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:17.043524 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:17.044414 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:17.044424 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:17.045108 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:17.045974 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:17.045999 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.047587 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.047592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:17.049309 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:17.049323 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.050435 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.050442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:17.050680 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:17.051476 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:17.051503 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:17.054163 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:17.058729 (+ 4566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:17.058738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:17.058754 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:17.059563 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:17.060089 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:17.060093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:17.060096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:17.060311 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:17.061413 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:17.061420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:17.061829 (+ 409us) client_negotiation.cc:769] Sending connection context
0809 22:37:17.062103 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 22:37:17.062805 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":509,"threads_started":1}
I20260809 22:37:17.066601 1852 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34135
I20260809 22:37:17.066962 1852 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:17.067664 1852 heartbeater.cc:511] Master 127.28.243.126:34135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:37:17.071240 1650 ts_manager.cc:194] Registered new tserver with Master: 22c40d8c5f8549c9b76f7a614213a223 (127.28.243.65:41503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
I20260809 22:37:17.073249 1650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:51393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:37:17.106366 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34135
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:17.462026 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:17.462404 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:17.462613 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:17.470567 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:17.470736 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:17.470917 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:17.481698 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:17.483155 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:17.484619 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:17.498440 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:17.498636 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:17.498685 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:17.499784 1858 server_base.cc:1034] running on GCE node
I20260809 22:37:17.500623 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:17.501735 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:17.503063 1858 hybrid_clock.cc:648] HybridClock initialized: now 1786315037502936 us; error 62 us; skew 500 ppm
I20260809 22:37:17.503484 1858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:17 dist-test-slave-rgb1 krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:17.507030 1858 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:37:17.508327 1858 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:17.508481 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:17.508873 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:17.511979 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "e59bfbe54db4405aa43e2be89dc35c9a"
format_stamp: "Formatted at 2026-08-09 22:37:17 on dist-test-slave-rgb1"
server_key: "d5162674dd1c5affde9c7672ff29b07e"
server_key_iv: "5d92fb5949ce4b3670f0443a77cbdf77"
server_key_version: "encryptionkey@0"
I20260809 22:37:17.512941 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "e59bfbe54db4405aa43e2be89dc35c9a"
format_stamp: "Formatted at 2026-08-09 22:37:17 on dist-test-slave-rgb1"
server_key: "d5162674dd1c5affde9c7672ff29b07e"
server_key_iv: "5d92fb5949ce4b3670f0443a77cbdf77"
server_key_version: "encryptionkey@0"
I20260809 22:37:17.518471 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 22:37:17.522330 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:17.523833 1858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 22:37:17.524046 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "e59bfbe54db4405aa43e2be89dc35c9a"
format_stamp: "Formatted at 2026-08-09 22:37:17 on dist-test-slave-rgb1"
server_key: "d5162674dd1c5affde9c7672ff29b07e"
server_key_iv: "5d92fb5949ce4b3670f0443a77cbdf77"
server_key_version: "encryptionkey@0"
I20260809 22:37:17.524312 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:17.552047 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:17.556809 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:17.557114 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:17.558274 1858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:17.559767 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:17.559860 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:17.559940 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:17.559998 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:17.598353 1858 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39335
I20260809 22:37:17.598490 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39335 every 8 connection(s)
I20260809 22:37:17.599287 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:17.603245 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 1858
I20260809 22:37:17.603487 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:17.604135 29645 external_mini_cluster.cc:1428] Setting key ff3c0c5ef73670d5f4b65c58d5039a54
Aug 09 22:37:17 dist-test-slave-rgb1 krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315037, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:17.624616 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:17.602157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:45513 (local address 127.28.243.126:34135)
0809 22:37:17.602843 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:17.602853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:17.603781 (+ 928us) server_negotiation.cc:407] Connection header received
0809 22:37:17.606369 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:17.606375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:17.606461 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:17.606599 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:17.609084 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.609822 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.612944 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.613432 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.613552 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:17.617963 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:17.617968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:17.617972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:17.618009 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:17.621275 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:17.622127 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:17.622132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:17.622134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:17.622286 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:17.622759 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:17.622769 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:17.622771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:17.623051 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:17.623218 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:17.623965 (+ 747us) server_negotiation.cc:299] Negotiation successful
0809 22:37:17.624206 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":284,"threads_started":1}
I20260809 22:37:17.625782 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:17.602603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34135 (local address 127.28.243.66:45513)
0809 22:37:17.603555 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:17.603623 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:17.605988 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:17.606912 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:17.606926 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:17.607853 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:17.608866 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:17.608899 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.610017 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.610027 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:17.612483 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:17.612506 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:17.613715 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:17.613725 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:17.614093 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:17.614944 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:17.614981 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:17.617670 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:17.621453 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:17.621463 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:17.621483 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:17.621971 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:17.622412 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:17.622418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:17.622422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:17.622628 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:17.623188 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:17.623197 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:17.623573 (+ 376us) client_negotiation.cc:769] Sending connection context
0809 22:37:17.623877 (+ 304us) client_negotiation.cc:240] Negotiation successful
0809 22:37:17.624448 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":349,"threads_started":1}
I20260809 22:37:17.627861 1994 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34135
I20260809 22:37:17.628216 1994 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:17.629181 1994 heartbeater.cc:511] Master 127.28.243.126:34135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
I20260809 22:37:17.631073 1650 ts_manager.cc:194] Registered new tserver with Master: e59bfbe54db4405aa43e2be89dc35c9a (127.28.243.66:39335)
I20260809 22:37:17.632128 1650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:45513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:37:17.665731 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34135
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:18.016119 2000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:18.016438 2000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:18.016671 2000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:18.025455 2000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:18.025592 2000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:18.025725 2000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:18.037493 2000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:18.038851 2000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:18.040335 2000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:18.054584 2005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:18.054817 2006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:18.054905 2008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:18.055717 2000 server_base.cc:1034] running on GCE node
I20260809 22:37:18.056463 2000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:18.057650 2000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:18.058876 2000 hybrid_clock.cc:648] HybridClock initialized: now 1786315038058827 us; error 43 us; skew 500 ppm
I20260809 22:37:18.059306 2000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:18 dist-test-slave-rgb1 krb5kdc[1602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315038, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:18.062976 2000 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:37:18.064455 2000 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:18.064638 2000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:18.065021 2000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:18.068269 2000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "441dd9b308484bd48fd52e94c3b93e13"
format_stamp: "Formatted at 2026-08-09 22:37:18 on dist-test-slave-rgb1"
server_key: "d0c5db63e325da595486da2a90d78348"
server_key_iv: "912eb6cb142775124ec5a5f15ba0d474"
server_key_version: "encryptionkey@0"
I20260809 22:37:18.069269 2000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "441dd9b308484bd48fd52e94c3b93e13"
format_stamp: "Formatted at 2026-08-09 22:37:18 on dist-test-slave-rgb1"
server_key: "d0c5db63e325da595486da2a90d78348"
server_key_iv: "912eb6cb142775124ec5a5f15ba0d474"
server_key_version: "encryptionkey@0"
I20260809 22:37:18.075459 2000 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:37:18.077116 1852 heartbeater.cc:503] Master 127.28.243.126:34135 was elected leader, sending a full tablet report...
I20260809 22:37:18.079974 2014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:18.081600 2000 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:18.081784 2000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "441dd9b308484bd48fd52e94c3b93e13"
format_stamp: "Formatted at 2026-08-09 22:37:18 on dist-test-slave-rgb1"
server_key: "d0c5db63e325da595486da2a90d78348"
server_key_iv: "912eb6cb142775124ec5a5f15ba0d474"
server_key_version: "encryptionkey@0"
I20260809 22:37:18.082048 2000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:18.106531 2000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:18.111742 2000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:18.112083 2000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:18.113204 2000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:18.115530 2000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:18.115685 2000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:18.115840 2000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:18.115937 2000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:18.156926 2000 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:40451
I20260809 22:37:18.156997 2131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:40451 every 8 connection(s)
I20260809 22:37:18.157909 2000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:18.162853 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2000
I20260809 22:37:18.163071 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:18.163640 29645 external_mini_cluster.cc:1428] Setting key faeff149c90ff0737eacf000bafda962
Aug 09 22:37:18 dist-test-slave-rgb1 krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315038, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:18.181588 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.160750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:56283 (local address 127.28.243.126:34135)
0809 22:37:18.161297 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.161302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.162219 (+ 917us) server_negotiation.cc:407] Connection header received
0809 22:37:18.164936 (+ 2717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.164942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.165059 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.165182 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:18.166785 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.167861 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.169905 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.170494 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.170710 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.174751 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:18.174756 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:18.174759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:18.174794 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:18.178063 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.178889 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.178901 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.178905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.179035 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.179505 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.179515 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.179517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.179874 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:18.180049 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.180921 (+ 872us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.181174 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":335,"threads_started":1}
I20260809 22:37:18.182559 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.161242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34135 (local address 127.28.243.67:56283)
0809 22:37:18.162048 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:18.162087 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:18.164567 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:18.165426 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:18.165434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:18.165968 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:18.166594 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.166612 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.168058 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.168067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.169738 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.169752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.170820 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.170828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.171080 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.171809 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:18.171834 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:18.174496 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:18.178217 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:18.178227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:18.178245 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:18.178745 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:18.179155 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:18.179160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:18.179164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:18.179372 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:18.180019 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:18.180028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:18.180475 (+ 447us) client_negotiation.cc:769] Sending connection context
0809 22:37:18.180843 (+ 368us) client_negotiation.cc:240] Negotiation successful
0809 22:37:18.181425 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260809 22:37:18.184149 2132 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34135
I20260809 22:37:18.184660 2132 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:18.185400 2132 heartbeater.cc:511] Master 127.28.243.126:34135 requested a full tablet report, sending...
I20260809 22:37:18.187026 1650 ts_manager.cc:194] Registered new tserver with Master: 441dd9b308484bd48fd52e94c3b93e13 (127.28.243.67:40451)
I20260809 22:37:18.187896 1650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:56283
I20260809 22:37:18.192598 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:37:18.212082 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.197708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.28.243.126:34135)
0809 22:37:18.197906 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.197911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.198273 (+ 362us) server_negotiation.cc:407] Connection header received
0809 22:37:18.198584 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.198592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.198669 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.198772 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:18.200464 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.201450 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.202897 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.203395 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.203513 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.206229 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:18.206234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:18.206237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:18.206275 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:18.209224 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.210015 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.210019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.210021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.210151 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.210522 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.210526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.210529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.210955 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:18.211166 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.211608 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.211809 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:37:18.217772 1650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:37:18.224973 1650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:18.251725 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.240889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.28.243.67:40451)
0809 22:37:18.242029 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.242054 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.242089 (+ 35us) server_negotiation.cc:407] Connection header received
0809 22:37:18.242186 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.242193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.242410 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.243401 (+ 991us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:18.243755 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.245610 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.248192 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.250050 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.250228 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.250919 (+ 691us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.251018 (+ 99us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.251219 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":791,"threads_started":1}
I20260809 22:37:18.252089 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.240176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:40451 (local address 127.0.0.1:57906)
0809 22:37:18.241244 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:18.241274 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:18.241406 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:18.242658 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:18.242663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:18.242695 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:18.243217 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.243230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.245777 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.245787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.247940 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.247959 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.250278 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.250284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.250507 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.250572 (+ 65us) client_negotiation.cc:769] Sending connection context
0809 22:37:18.250711 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 22:37:18.250928 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":28,"thread_start_us":1014,"threads_started":1}
I20260809 22:37:18.252334 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.238631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.28.243.66:39335)
0809 22:37:18.240115 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.240127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.240161 (+ 34us) server_negotiation.cc:407] Connection header received
0809 22:37:18.240276 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.240287 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.240638 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.241805 (+ 1167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:18.242175 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.244092 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.248702 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.250368 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.251533 (+ 1165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.251727 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.251783 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.251992 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1292,"thread_start_us":1119,"threads_started":1}
I20260809 22:37:18.253065 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.240671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:41503 (local address 127.0.0.1:41868)
0809 22:37:18.242844 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:18.242870 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:18.242996 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:18.244118 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:18.244126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:18.244159 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:18.244788 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.244810 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.247196 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.247203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.249452 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.249466 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.251347 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.251354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.252206 (+ 852us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.252252 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 22:37:18.252352 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 22:37:18.252528 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2027,"thread_start_us":1873,"threads_started":1}
I20260809 22:37:18.253350 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.238411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39335 (local address 127.0.0.1:45692)
0809 22:37:18.239249 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:18.239289 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:18.239496 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:18.240975 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:18.240984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:18.241036 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:18.241559 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.241576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.244318 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.244323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.248104 (+ 3781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.248124 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.250663 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.250674 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.250937 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.250982 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 22:37:18.251064 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 22:37:18.251228 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":1,"thread_start_us":440,"threads_started":1}
I20260809 22:37:18.254020 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.241436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.28.243.65:41503)
0809 22:37:18.242350 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.242359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.243141 (+ 782us) server_negotiation.cc:407] Connection header received
0809 22:37:18.243248 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.243256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.243477 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.243682 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:18.245033 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.246952 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.249629 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.250836 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.250962 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.251186 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.253363 (+ 2177us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.253571 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":570,"threads_started":1}
I20260809 22:37:18.257018 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 20c12a6cde3e469b9e368ee5e461e0a4 (DEFAULT_TABLE table=test-table [id=860fb87430d74713a883945e8829a165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:18.257079 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 20c12a6cde3e469b9e368ee5e461e0a4 (DEFAULT_TABLE table=test-table [id=860fb87430d74713a883945e8829a165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:18.257079 2065 tablet_service.cc:1467] Processing CreateTablet for tablet 20c12a6cde3e469b9e368ee5e461e0a4 (DEFAULT_TABLE table=test-table [id=860fb87430d74713a883945e8829a165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:18.258680 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c12a6cde3e469b9e368ee5e461e0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:18.258690 2065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c12a6cde3e469b9e368ee5e461e0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:18.258699 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c12a6cde3e469b9e368ee5e461e0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:18.271411 2151 tablet_bootstrap.cc:492] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Bootstrap starting.
I20260809 22:37:18.275916 2151 tablet_bootstrap.cc:654] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:18.277015 2153 tablet_bootstrap.cc:492] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Bootstrap starting.
I20260809 22:37:18.277544 2152 tablet_bootstrap.cc:492] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Bootstrap starting.
I20260809 22:37:18.277510 2151 log.cc:824] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:18.281251 2151 tablet_bootstrap.cc:492] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: No bootstrap required, opened a new log
I20260809 22:37:18.281543 2151 ts_tablet_manager.cc:1397] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 22:37:18.281664 2153 tablet_bootstrap.cc:654] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:18.282671 2152 tablet_bootstrap.cc:654] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:18.282902 2153 log.cc:824] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:18.283973 2152 log.cc:824] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:18.286484 2153 tablet_bootstrap.cc:492] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: No bootstrap required, opened a new log
I20260809 22:37:18.286937 2153 ts_tablet_manager.cc:1397] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260809 22:37:18.294766 2152 tablet_bootstrap.cc:492] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: No bootstrap required, opened a new log
I20260809 22:37:18.294466 2151 raft_consensus.cc:348] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.294973 2151 raft_consensus.cc:374] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:18.295068 2151 raft_consensus.cc:729] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c40d8c5f8549c9b76f7a614213a223, State: Initialized, Role: FOLLOWER
I20260809 22:37:18.295127 2152 ts_tablet_manager.cc:1397] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20260809 22:37:18.295681 2151 consensus_queue.cc:260] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.297032 2153 raft_consensus.cc:348] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.297542 2153 raft_consensus.cc:374] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:18.297569 2151 ts_tablet_manager.cc:1428] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260809 22:37:18.297655 2153 raft_consensus.cc:729] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59bfbe54db4405aa43e2be89dc35c9a, State: Initialized, Role: FOLLOWER
I20260809 22:37:18.298457 2153 consensus_queue.cc:260] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.299809 1994 heartbeater.cc:503] Master 127.28.243.126:34135 was elected leader, sending a full tablet report...
I20260809 22:37:18.300655 2153 ts_tablet_manager.cc:1428] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260809 22:37:18.306849 2152 raft_consensus.cc:348] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.307389 2152 raft_consensus.cc:374] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:18.307488 2152 raft_consensus.cc:729] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 441dd9b308484bd48fd52e94c3b93e13, State: Initialized, Role: FOLLOWER
I20260809 22:37:18.308102 2152 consensus_queue.cc:260] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.309342 2132 heartbeater.cc:503] Master 127.28.243.126:34135 was elected leader, sending a full tablet report...
I20260809 22:37:18.310243 2152 ts_tablet_manager.cc:1428] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
W20260809 22:37:18.351306 1995 tablet.cc:2369] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:18.369712 2159 raft_consensus.cc:482] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:18.370080 2159 raft_consensus.cc:504] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.371915 2159 leader_election.cc:288] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59bfbe54db4405aa43e2be89dc35c9a (127.28.243.66:39335), 22c40d8c5f8549c9b76f7a614213a223 (127.28.243.65:41503)
I20260809 22:37:18.379864 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.372522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:43059 (local address 127.28.243.66:39335)
0809 22:37:18.372939 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.372947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.372971 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:37:18.373074 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.373082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.373181 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.373296 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:18.373939 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.375357 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.377673 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.378865 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.378987 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.379043 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.379483 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.379603 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260809 22:37:18.380018 2135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.372101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39335 (local address 127.28.243.67:43059)
0809 22:37:18.372319 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:18.372348 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:18.372495 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:18.373357 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:18.373360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:18.373421 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:18.373737 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.373749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.375538 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.375545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.377473 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.377486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.379043 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.379050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.379271 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.379315 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 22:37:18.379453 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 22:37:18.379617 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"spinlock_wait_cycles":25472}
I20260809 22:37:18.381189 2161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.372462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:41503 (local address 127.28.243.67:59675)
0809 22:37:18.373313 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:18.373333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:18.373537 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:18.373939 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:18.373944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:18.373972 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:18.374658 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.374677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.376507 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.376511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.378424 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:18.378440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.380477 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.380482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:18.380673 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.380697 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 22:37:18.380829 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 22:37:18.380942 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":467,"threads_started":1}
I20260809 22:37:18.381305 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.372496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:59675 (local address 127.28.243.65:41503)
0809 22:37:18.372763 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.372771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.373491 (+ 720us) server_negotiation.cc:407] Connection header received
0809 22:37:18.373689 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.373694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.373774 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.373865 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:18.374960 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.376262 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.378625 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.380133 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.380482 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.380604 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.380863 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.381038 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 22:37:18.381488 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c12a6cde3e469b9e368ee5e461e0a4" candidate_uuid: "441dd9b308484bd48fd52e94c3b93e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" is_pre_election: true
I20260809 22:37:18.382149 1945 raft_consensus.cc:2454] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441dd9b308484bd48fd52e94c3b93e13 in term 0.
I20260809 22:37:18.382398 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c12a6cde3e469b9e368ee5e461e0a4" candidate_uuid: "441dd9b308484bd48fd52e94c3b93e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c40d8c5f8549c9b76f7a614213a223" is_pre_election: true
I20260809 22:37:18.382887 1803 raft_consensus.cc:2454] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 441dd9b308484bd48fd52e94c3b93e13 in term 0.
I20260809 22:37:18.382959 2016 leader_election.cc:302] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 441dd9b308484bd48fd52e94c3b93e13, e59bfbe54db4405aa43e2be89dc35c9a; no voters:
I20260809 22:37:18.383366 2159 raft_consensus.cc:2781] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:18.383535 2159 raft_consensus.cc:482] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:18.383708 2159 raft_consensus.cc:3037] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:18.386490 2159 raft_consensus.cc:504] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.387300 2159 leader_election.cc:288] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [CANDIDATE]: Term 1 election: Requested vote from peers e59bfbe54db4405aa43e2be89dc35c9a (127.28.243.66:39335), 22c40d8c5f8549c9b76f7a614213a223 (127.28.243.65:41503)
I20260809 22:37:18.387812 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c12a6cde3e469b9e368ee5e461e0a4" candidate_uuid: "441dd9b308484bd48fd52e94c3b93e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59bfbe54db4405aa43e2be89dc35c9a"
I20260809 22:37:18.387836 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c12a6cde3e469b9e368ee5e461e0a4" candidate_uuid: "441dd9b308484bd48fd52e94c3b93e13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c40d8c5f8549c9b76f7a614213a223"
I20260809 22:37:18.388077 1945 raft_consensus.cc:3037] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:18.388087 1803 raft_consensus.cc:3037] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:18.390533 1803 raft_consensus.cc:2454] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441dd9b308484bd48fd52e94c3b93e13 in term 1.
I20260809 22:37:18.390609 1945 raft_consensus.cc:2454] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 441dd9b308484bd48fd52e94c3b93e13 in term 1.
I20260809 22:37:18.391069 2015 leader_election.cc:302] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c40d8c5f8549c9b76f7a614213a223, 441dd9b308484bd48fd52e94c3b93e13; no voters:
I20260809 22:37:18.391498 2159 raft_consensus.cc:2781] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:18.392000 2159 raft_consensus.cc:686] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 1 LEADER]: Becoming Leader. State: Replica: 441dd9b308484bd48fd52e94c3b93e13, State: Running, Role: LEADER
I20260809 22:37:18.392699 2159 consensus_queue.cc:237] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } }
I20260809 22:37:18.398439 1650 catalog_manager.cc:5495] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 441dd9b308484bd48fd52e94c3b93e13 (127.28.243.67). New cstate: current_term: 1 leader_uuid: "441dd9b308484bd48fd52e94c3b93e13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "441dd9b308484bd48fd52e94c3b93e13" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 } health_report { overall_health: UNKNOWN } } }
W20260809 22:37:18.409698 2133 tablet.cc:2369] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:18.449596 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.443297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.28.243.67:40451)
0809 22:37:18.443621 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.443631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.443706 (+ 75us) server_negotiation.cc:407] Connection header received
0809 22:37:18.444003 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.444010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.444137 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.444323 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:18.444817 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.445623 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.447116 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.447577 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.447730 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.448295 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:18.448924 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:18.449103 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.449194 (+ 91us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.449284 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 22:37:18.458712 1945 raft_consensus.cc:1261] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 1 FOLLOWER]: Refusing update from remote peer 441dd9b308484bd48fd52e94c3b93e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:18.458710 1803 raft_consensus.cc:1261] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 1 FOLLOWER]: Refusing update from remote peer 441dd9b308484bd48fd52e94c3b93e13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:18.460076 2162 consensus_queue.cc:1035] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59bfbe54db4405aa43e2be89dc35c9a" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:18.460496 2159 consensus_queue.cc:1035] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c40d8c5f8549c9b76f7a614213a223" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:18.471733 2165 mvcc.cc:205] Tried to move back new op lower bound from 7316746397515321344 to 7316746397261160448. Current Snapshot: MvccSnapshot[applied={T|T < 7316746397515321344}]
I20260809 22:37:18.472098 2167 mvcc.cc:205] Tried to move back new op lower bound from 7316746397515321344 to 7316746397261160448. Current Snapshot: MvccSnapshot[applied={T|T < 7316746397515321344}]
I20260809 22:37:18.482059 2166 mvcc.cc:205] Tried to move back new op lower bound from 7316746397515321344 to 7316746397261160448. Current Snapshot: MvccSnapshot[applied={T|T < 7316746397515321344}]
I20260809 22:37:18.491873 1648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:18.492278 1648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:18.496410 1648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25bc1f6e07184d0582314f19d9d3526a: Sending DeleteTablet for 3 replicas of tablet 20c12a6cde3e469b9e368ee5e461e0a4
I20260809 22:37:18.497867 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c12a6cde3e469b9e368ee5e461e0a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:18 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:45692
I20260809 22:37:18.497979 2065 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c12a6cde3e469b9e368ee5e461e0a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:18 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:57906
I20260809 22:37:18.498728 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c12a6cde3e469b9e368ee5e461e0a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:18 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:41868
I20260809 22:37:18.501380 2175 tablet_replica.cc:330] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: stopping tablet replica
I20260809 22:37:18.501916 2175 raft_consensus.cc:2229] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:18.502696 2175 raft_consensus.cc:2258] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:18.502923 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 1716
I20260809 22:37:18.505089 2175 ts_tablet_manager.cc:1905] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:18.507931 2177 tablet_replica.cc:330] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: stopping tablet replica
I20260809 22:37:18.508512 2177 raft_consensus.cc:2229] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:18.509186 2177 raft_consensus.cc:2258] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:18.513343 2175 ts_tablet_manager.cc:1918] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:18.513610 2175 log.cc:1196] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/20c12a6cde3e469b9e368ee5e461e0a4
I20260809 22:37:18.514277 2175 ts_tablet_manager.cc:1939] T 20c12a6cde3e469b9e368ee5e461e0a4 P 441dd9b308484bd48fd52e94c3b93e13: Deleting consensus metadata
I20260809 22:37:18.515595 2176 tablet_replica.cc:330] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: stopping tablet replica
I20260809 22:37:18.516085 2176 raft_consensus.cc:2229] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:18.516435 1636 catalog_manager.cc:4841] TS 441dd9b308484bd48fd52e94c3b93e13 (127.28.243.67:40451): tablet 20c12a6cde3e469b9e368ee5e461e0a4 (table test-table [id=860fb87430d74713a883945e8829a165]) successfully deleted
I20260809 22:37:18.516664 2176 raft_consensus.cc:2258] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:18.518649 2177 ts_tablet_manager.cc:1905] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:18.519392 2176 ts_tablet_manager.cc:1905] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:18.526381 2177 ts_tablet_manager.cc:1918] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:18.526671 2177 log.cc:1196] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/20c12a6cde3e469b9e368ee5e461e0a4
I20260809 22:37:18.526710 2176 ts_tablet_manager.cc:1918] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:18.526911 2176 log.cc:1196] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/20c12a6cde3e469b9e368ee5e461e0a4
I20260809 22:37:18.527388 2177 ts_tablet_manager.cc:1939] T 20c12a6cde3e469b9e368ee5e461e0a4 P 22c40d8c5f8549c9b76f7a614213a223: Deleting consensus metadata
I20260809 22:37:18.527520 2176 ts_tablet_manager.cc:1939] T 20c12a6cde3e469b9e368ee5e461e0a4 P e59bfbe54db4405aa43e2be89dc35c9a: Deleting consensus metadata
Aug 09 22:37:18 dist-test-slave-rgb1 krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:37:18.529597 1634 catalog_manager.cc:4841] TS e59bfbe54db4405aa43e2be89dc35c9a (127.28.243.66:39335): tablet 20c12a6cde3e469b9e368ee5e461e0a4 (table test-table [id=860fb87430d74713a883945e8829a165]) successfully deleted
I20260809 22:37:18.530192 1633 catalog_manager.cc:4841] TS 22c40d8c5f8549c9b76f7a614213a223 (127.28.243.65:41503): tablet 20c12a6cde3e469b9e368ee5e461e0a4 (table test-table [id=860fb87430d74713a883945e8829a165]) successfully deleted
I20260809 22:37:18.535470 2150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.504717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.28.243.65:41503)
0809 22:37:18.504970 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.504980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.515994 (+ 11014us) server_negotiation.cc:407] Connection header received
0809 22:37:18.516523 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.516529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.516662 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.516747 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:18.518947 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.520060 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.521968 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.522607 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.522717 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.529089 (+ 6372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:18.529107 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:18.529126 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:18.529177 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:18.532428 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.533218 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.533227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.533229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.533380 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.533781 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.533788 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.533795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.534180 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:18.534515 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.534832 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.535106 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 22:37:18.536195 1847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:18.665832 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 1716
I20260809 22:37:18.685685 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 1858
Aug 09 22:37:18 dist-test-slave-rgb1 krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:37:18.703689 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.686641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.28.243.66:39335)
0809 22:37:18.686821 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.686825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.686865 (+ 40us) server_negotiation.cc:407] Connection header received
0809 22:37:18.687123 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.687129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.687211 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.687305 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:18.689017 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.689786 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.691612 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.692088 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.692187 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.697524 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:18.697541 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:18.697557 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:18.697600 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:18.700792 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.701538 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.701546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.701548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.701688 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.702077 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.702083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.702086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.702484 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:18.702780 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.703111 (+ 331us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.703340 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 22:37:18.704408 1989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:18.797230 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 1858
I20260809 22:37:18.823983 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2000
Aug 09 22:37:18 dist-test-slave-rgb1 krb5kdc[1602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:37:18.845129 2149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:18.824893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.28.243.67:40451)
0809 22:37:18.825110 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:18.825116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:18.825142 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:37:18.825240 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:18.825245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:18.825331 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:18.825469 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:18.827194 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.828420 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.830344 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:18.831135 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:18.831325 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:18.836207 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:18.836230 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:18.836252 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:18.836314 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:18.841243 (+ 4929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.842250 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.842262 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.842267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.842512 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:18.842997 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:18.843003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:18.843007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:18.843529 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:18.843951 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:18.844308 (+ 357us) server_negotiation.cc:299] Negotiation successful
0809 22:37:18.844601 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 22:37:18.845980 2127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:18.975538 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2000
I20260809 22:37:18.998245 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 1616
I20260809 22:37:18.999589 1683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:19.114852 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 1616
2026-08-09T22:37:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4769 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T22:37:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:19Z Disabled control of system clock
I20260809 22:37:19.199719 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39713
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 22:37:19.570474 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:19.570844 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:19.571019 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:19.579106 2188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:19.579221 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:19.579310 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:19.579398 2188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:19.579439 2188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:19.590327 2188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:19.591876 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:19.593950 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:19.607983 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:19.608103 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:19.607944 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:19.609350 2188 server_base.cc:1034] running on GCE node
I20260809 22:37:19.610314 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:19.611954 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:19.613214 2188 hybrid_clock.cc:648] HybridClock initialized: now 1786315039613150 us; error 51 us; skew 500 ppm
I20260809 22:37:19.613658 2188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:19.616498 2188 webserver.cc:460] Webserver started at http://127.28.243.126:46503/ using document root <none> and password file <none>
I20260809 22:37:19.617424 2188 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:19.617533 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:19.617833 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:19.621310 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "d4f49abef7ae4b77af46ace5e5ebf958"
format_stamp: "Formatted at 2026-08-09 22:37:19 on dist-test-slave-rgb1"
server_key: "82604d37ddd2ca21e0362fa0bdca4ecd"
server_key_iv: "0b3a190658f15f6b532f64dee1091404"
server_key_version: "encryptionkey@0"
I20260809 22:37:19.622164 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "d4f49abef7ae4b77af46ace5e5ebf958"
format_stamp: "Formatted at 2026-08-09 22:37:19 on dist-test-slave-rgb1"
server_key: "82604d37ddd2ca21e0362fa0bdca4ecd"
server_key_iv: "0b3a190658f15f6b532f64dee1091404"
server_key_version: "encryptionkey@0"
I20260809 22:37:19.628170 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260809 22:37:19.632320 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:19.634150 2188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:19.634461 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "d4f49abef7ae4b77af46ace5e5ebf958"
format_stamp: "Formatted at 2026-08-09 22:37:19 on dist-test-slave-rgb1"
server_key: "82604d37ddd2ca21e0362fa0bdca4ecd"
server_key_iv: "0b3a190658f15f6b532f64dee1091404"
server_key_version: "encryptionkey@0"
I20260809 22:37:19.634805 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:19.641767 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:19.642700 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:19.643064 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:19.664774 2188 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:39713
I20260809 22:37:19.664800 2255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:39713 every 8 connection(s)
I20260809 22:37:19.666381 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:19.670698 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:19.675590 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2188
I20260809 22:37:19.675808 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:19.676497 29645 external_mini_cluster.cc:1428] Setting key a84a671df7f8e00bca1c058a97e064e7
I20260809 22:37:19.681986 2257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958: Bootstrap starting.
I20260809 22:37:19.685794 2257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:19.686205 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:19.678926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.28.243.126:39713)
0809 22:37:19.680254 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:19.680266 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:19.680311 (+ 45us) server_negotiation.cc:407] Connection header received
0809 22:37:19.682213 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:19.682249 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:19.682717 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:19.683072 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:19.684335 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:19.684342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:19.684355 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:19.684371 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:19.684475 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:19.684494 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:19.684764 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:19.684869 (+ 105us) server_negotiation.cc:299] Negotiation successful
0809 22:37:19.685160 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":477,"threads_started":1}
I20260809 22:37:19.687171 2257 log.cc:824] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:19.690659 2257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958: No bootstrap required, opened a new log
I20260809 22:37:19.698196 2257 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39713 } }
I20260809 22:37:19.698596 2257 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:19.698722 2257 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f49abef7ae4b77af46ace5e5ebf958, State: Initialized, Role: FOLLOWER
I20260809 22:37:19.699374 2257 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39713 } }
I20260809 22:37:19.699636 2257 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:19.699786 2257 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:19.699971 2257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:19.702348 2257 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39713 } }
I20260809 22:37:19.702929 2257 leader_election.cc:302] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f49abef7ae4b77af46ace5e5ebf958; no voters:
I20260809 22:37:19.703606 2257 leader_election.cc:288] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:19.703795 2263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:19.704643 2263 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [term 1 LEADER]: Becoming Leader. State: Replica: d4f49abef7ae4b77af46ace5e5ebf958, State: Running, Role: LEADER
I20260809 22:37:19.705222 2263 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39713 } }
I20260809 22:37:19.705950 2257 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:19.708802 2264 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39713 } } }
I20260809 22:37:19.708778 2265 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f49abef7ae4b77af46ace5e5ebf958. Latest consensus state: current_term: 1 leader_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f49abef7ae4b77af46ace5e5ebf958" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39713 } } }
I20260809 22:37:19.709278 2264 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:19.709362 2265 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:19.710271 2272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:19.718345 2272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:19.728814 2272 catalog_manager.cc:1347] Generated new cluster ID: 25c51f600e034cef9b8e07b813891235
I20260809 22:37:19.728965 2272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:19.746788 2272 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:19.747483 2272 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:19.761770 2272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958: Generated new TSK 0
I20260809 22:37:19.762884 2272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:37:19.780915 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39713
--builtin_ntp_servers=127.28.243.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 22:37:20.184912 2283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:20.185299 2283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:20.185482 2283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:20.193241 2283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:20.193394 2283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:20.193584 2283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:20.203250 2283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:20.204713 2283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:20.206646 2283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:20.221140 2288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:20.221076 2289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:20.221535 2291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:20.222371 2283 server_base.cc:1034] running on GCE node
I20260809 22:37:20.223119 2283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:20.224299 2283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:20.225589 2283 hybrid_clock.cc:648] HybridClock initialized: now 1786315040225518 us; error 91 us; skew 500 ppm
I20260809 22:37:20.225983 2283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:20.228842 2283 webserver.cc:460] Webserver started at http://127.28.243.65:35455/ using document root <none> and password file <none>
I20260809 22:37:20.229612 2283 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:20.229756 2283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:20.230125 2283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:20.233196 2283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "437f8a900d5d46e0ae02c6187542c111"
format_stamp: "Formatted at 2026-08-09 22:37:20 on dist-test-slave-rgb1"
server_key: "f9c2951b02902644b85c4658ed8ad2bf"
server_key_iv: "ffa3b5b8978965da262b092bfc745478"
server_key_version: "encryptionkey@0"
I20260809 22:37:20.234066 2283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "437f8a900d5d46e0ae02c6187542c111"
format_stamp: "Formatted at 2026-08-09 22:37:20 on dist-test-slave-rgb1"
server_key: "f9c2951b02902644b85c4658ed8ad2bf"
server_key_iv: "ffa3b5b8978965da262b092bfc745478"
server_key_version: "encryptionkey@0"
I20260809 22:37:20.239756 2283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 22:37:20.243994 2299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:20.246093 2283 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:20.246325 2283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "437f8a900d5d46e0ae02c6187542c111"
format_stamp: "Formatted at 2026-08-09 22:37:20 on dist-test-slave-rgb1"
server_key: "f9c2951b02902644b85c4658ed8ad2bf"
server_key_iv: "ffa3b5b8978965da262b092bfc745478"
server_key_version: "encryptionkey@0"
I20260809 22:37:20.246651 2283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:20.253455 2283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:20.254420 2283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:20.254753 2283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:20.255939 2283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:20.257727 2283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:20.257879 2283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:20.258030 2283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:20.258143 2283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:20.298377 2283 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:36309
I20260809 22:37:20.298470 2419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:36309 every 8 connection(s)
I20260809 22:37:20.299774 2283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:20.302259 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2283
I20260809 22:37:20.302524 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:20.303339 29645 external_mini_cluster.cc:1428] Setting key d3e8bf3128ba0c6e92766c72c7a0f895
I20260809 22:37:20.306659 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39713
--builtin_ntp_servers=127.28.243.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 22:37:20.309967 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:20.303221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:59855 (local address 127.28.243.126:39713)
0809 22:37:20.303886 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:20.303893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:20.305258 (+ 1365us) server_negotiation.cc:407] Connection header received
0809 22:37:20.307453 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:20.307460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:20.307551 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:20.307678 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:20.309135 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:20.309141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:20.309146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:20.309166 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:20.309220 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:20.309232 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:20.309368 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:20.309571 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 22:37:20.309646 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":365,"threads_started":1}
I20260809 22:37:20.311331 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:20.303913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39713 (local address 127.28.243.65:59855)
0809 22:37:20.305043 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:20.305085 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:20.307176 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:20.307926 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:20.307935 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:20.308042 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:20.308823 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:20.308826 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:20.308866 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:20.308954 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:20.309331 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:20.309343 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:20.309353 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 22:37:20.309659 (+ 306us) client_negotiation.cc:240] Negotiation successful
0809 22:37:20.309926 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":380,"threads_started":1}
I20260809 22:37:20.313364 2421 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39713
I20260809 22:37:20.313752 2421 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:20.314545 2421 heartbeater.cc:511] Master 127.28.243.126:39713 requested a full tablet report, sending...
I20260809 22:37:20.316934 2220 ts_manager.cc:194] Registered new tserver with Master: 437f8a900d5d46e0ae02c6187542c111 (127.28.243.65:36309)
W20260809 22:37:20.664249 2425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:20.664655 2425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:20.664790 2425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:20.676513 2425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:20.676671 2425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:20.676847 2425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:20.690773 2425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:20.692054 2425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:20.694284 2425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:20.708690 2430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:20.708745 2431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:20.708690 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:20.710165 2425 server_base.cc:1034] running on GCE node
I20260809 22:37:20.710932 2425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:20.712142 2425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:20.713368 2425 hybrid_clock.cc:648] HybridClock initialized: now 1786315040713312 us; error 50 us; skew 500 ppm
I20260809 22:37:20.713769 2425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:20.716827 2425 webserver.cc:460] Webserver started at http://127.28.243.66:38907/ using document root <none> and password file <none>
I20260809 22:37:20.717613 2425 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:20.717713 2425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:20.718154 2425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:20.721395 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d"
format_stamp: "Formatted at 2026-08-09 22:37:20 on dist-test-slave-rgb1"
server_key: "be5a4a67159830496237e8dfb379fa52"
server_key_iv: "670ace1ff2be581d8ba2827b10917593"
server_key_version: "encryptionkey@0"
I20260809 22:37:20.722275 2425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d"
format_stamp: "Formatted at 2026-08-09 22:37:20 on dist-test-slave-rgb1"
server_key: "be5a4a67159830496237e8dfb379fa52"
server_key_iv: "670ace1ff2be581d8ba2827b10917593"
server_key_version: "encryptionkey@0"
I20260809 22:37:20.727958 2425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 22:37:20.731806 2439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:20.733459 2425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:20.733649 2425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d"
format_stamp: "Formatted at 2026-08-09 22:37:20 on dist-test-slave-rgb1"
server_key: "be5a4a67159830496237e8dfb379fa52"
server_key_iv: "670ace1ff2be581d8ba2827b10917593"
server_key_version: "encryptionkey@0"
I20260809 22:37:20.733892 2425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:20.740379 2425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:20.741370 2425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:20.741676 2425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:20.742687 2425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:20.744324 2425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:20.744463 2425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:20.744644 2425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:20.744741 2425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 22:37:20.783320 2425 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:44209
I20260809 22:37:20.783416 2560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:44209 every 8 connection(s)
I20260809 22:37:20.784727 2425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:20.793323 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:20.787669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:47411 (local address 127.28.243.126:39713)
0809 22:37:20.787902 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:20.787907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:20.789502 (+ 1595us) server_negotiation.cc:407] Connection header received
0809 22:37:20.791229 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:20.791234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:20.791292 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:20.791388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:20.792711 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:20.792716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:20.792719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:20.792735 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:20.792769 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:20.792781 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:20.792921 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:20.793051 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 22:37:20.793106 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:37:20.794291 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:20.788265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39713 (local address 127.28.243.66:47411)
0809 22:37:20.789342 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:20.789378 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:20.790930 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:20.791588 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:20.791596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:20.791687 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:20.792368 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:20.792370 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:20.792407 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:20.792489 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:20.792873 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:20.792880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:20.792886 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 22:37:20.793106 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 22:37:20.793294 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":378,"threads_started":1}
I20260809 22:37:20.794593 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2425
I20260809 22:37:20.794777 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:20.795365 29645 external_mini_cluster.cc:1428] Setting key 9470604d3fb21a63481dc2f59953d078
I20260809 22:37:20.796344 2561 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39713
I20260809 22:37:20.796720 2561 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:20.797366 2561 heartbeater.cc:511] Master 127.28.243.126:39713 requested a full tablet report, sending...
I20260809 22:37:20.798797 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39713
--builtin_ntp_servers=127.28.243.84:41447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 22:37:20.799144 2220 ts_manager.cc:194] Registered new tserver with Master: f4369e9938fb42fa9c2fb4a2a2d8656d (127.28.243.66:44209)
W20260809 22:37:21.160943 2564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:21.161432 2564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:21.161608 2564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:21.169548 2564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:21.169677 2564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:21.169865 2564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:21.179670 2564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39713
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:21.181169 2564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:21.182958 2564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:21.198014 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:21.198102 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:21.198323 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:21.200294 2564 server_base.cc:1034] running on GCE node
I20260809 22:37:21.201102 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:21.202253 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:21.203496 2564 hybrid_clock.cc:648] HybridClock initialized: now 1786315041203444 us; error 40 us; skew 500 ppm
I20260809 22:37:21.203940 2564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:21.207219 2564 webserver.cc:460] Webserver started at http://127.28.243.67:44247/ using document root <none> and password file <none>
I20260809 22:37:21.208118 2564 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:21.208253 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:21.208752 2564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:21.212008 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "a41670ae2de944059515013c58688770"
format_stamp: "Formatted at 2026-08-09 22:37:21 on dist-test-slave-rgb1"
server_key: "270551f7740508a0a0630cfd398326c0"
server_key_iv: "fc9e371af1981ad6fe9072872b60851b"
server_key_version: "encryptionkey@0"
I20260809 22:37:21.213027 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "a41670ae2de944059515013c58688770"
format_stamp: "Formatted at 2026-08-09 22:37:21 on dist-test-slave-rgb1"
server_key: "270551f7740508a0a0630cfd398326c0"
server_key_iv: "fc9e371af1981ad6fe9072872b60851b"
server_key_version: "encryptionkey@0"
I20260809 22:37:21.218966 2564 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:37:21.223030 2578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:21.224493 2564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:37:21.224768 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "a41670ae2de944059515013c58688770"
format_stamp: "Formatted at 2026-08-09 22:37:21 on dist-test-slave-rgb1"
server_key: "270551f7740508a0a0630cfd398326c0"
server_key_iv: "fc9e371af1981ad6fe9072872b60851b"
server_key_version: "encryptionkey@0"
I20260809 22:37:21.225083 2564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:21.231560 2564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:21.232525 2564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:21.232906 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:21.233973 2564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:21.235635 2564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:21.235777 2564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:21.235929 2564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:21.236048 2564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:21.275507 2564 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:46687
I20260809 22:37:21.275617 2697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:46687 every 8 connection(s)
I20260809 22:37:21.277127 2564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:21.285897 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.279957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:52687 (local address 127.28.243.126:39713)
0809 22:37:21.280187 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.280193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.281703 (+ 1510us) server_negotiation.cc:407] Connection header received
0809 22:37:21.283393 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.283397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.283455 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.283560 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.285211 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.285216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.285219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.285238 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.285272 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.285284 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.285394 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.285611 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.285668 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 22:37:21.287052 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.280506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39713 (local address 127.28.243.67:52687)
0809 22:37:21.281542 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.281579 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:21.283129 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:21.283836 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:21.283844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:21.284012 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:21.284825 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:21.284828 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:21.284869 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:21.285011 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:21.285412 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:21.285420 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:21.285427 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 22:37:21.285667 (+ 240us) client_negotiation.cc:240] Negotiation successful
0809 22:37:21.285887 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":338,"threads_started":1}
I20260809 22:37:21.287427 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2564
I20260809 22:37:21.287657 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:21.288144 29645 external_mini_cluster.cc:1428] Setting key 0d2f7bdd5e2f228a8a4926d713a90cea
I20260809 22:37:21.288785 2698 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39713
I20260809 22:37:21.289206 2698 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:21.289994 2698 heartbeater.cc:511] Master 127.28.243.126:39713 requested a full tablet report, sending...
I20260809 22:37:21.291836 2220 ts_manager.cc:194] Registered new tserver with Master: a41670ae2de944059515013c58688770 (127.28.243.67:46687)
I20260809 22:37:21.304461 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:37:21.312814 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.309966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.28.243.126:39713)
0809 22:37:21.310280 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.310288 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.310622 (+ 334us) server_negotiation.cc:407] Connection header received
0809 22:37:21.310844 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.310848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.310929 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.311054 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.312057 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.312063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.312067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.312092 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.312124 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.312137 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.312248 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.312402 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.312451 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260809 22:37:21.320011 2421 heartbeater.cc:503] Master 127.28.243.126:39713 was elected leader, sending a full tablet report...
I20260809 22:37:21.317252 2220 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:37:21.324859 2220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:21.345729 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.340050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.28.243.67:46687)
0809 22:37:21.341859 (+ 1809us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.341868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.341904 (+ 36us) server_negotiation.cc:407] Connection header received
0809 22:37:21.341993 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.342004 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.342244 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.342406 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.344671 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.344679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.344691 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.344710 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.344767 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.344786 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.344961 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.345183 (+ 222us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.345284 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":1433,"threads_started":1}
I20260809 22:37:21.345728 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.340761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:44209 (local address 127.0.0.1:43986)
0809 22:37:21.342566 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.342589 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:21.342718 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:21.343589 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:21.343595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:21.343619 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:21.344385 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:21.344388 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:21.344403 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:21.344441 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:21.344891 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:21.344903 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:21.344959 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 22:37:21.345183 (+ 224us) client_negotiation.cc:240] Negotiation successful
0809 22:37:21.345282 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1051,"thread_start_us":322,"threads_started":1}
I20260809 22:37:21.346396 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.341110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.28.243.66:44209)
0809 22:37:21.342337 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.342348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.342860 (+ 512us) server_negotiation.cc:407] Connection header received
0809 22:37:21.342946 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.342957 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.343194 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.343357 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.344670 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.344677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.344688 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.344703 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.344754 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.344774 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.344932 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.345994 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.346092 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":927,"threads_started":1}
I20260809 22:37:21.346961 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.339409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:46687 (local address 127.0.0.1:54510)
0809 22:37:21.341341 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.341364 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:21.341461 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:21.343020 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:21.343030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:21.343132 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:21.344036 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:21.344040 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:21.344072 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:21.344168 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:21.345003 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:21.345007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:21.345011 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 22:37:21.346430 (+ 1419us) client_negotiation.cc:240] Negotiation successful
0809 22:37:21.346494 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1812,"thread_start_us":2281,"threads_started":1}
I20260809 22:37:21.347893 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.338861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35536)
0809 22:37:21.340413 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.340451 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:21.340615 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:21.345084 (+ 4469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:21.345088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:21.345109 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:21.346742 (+ 1633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:21.346746 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:21.346763 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:21.346815 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:21.347303 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:21.347308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:21.347313 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 22:37:21.347452 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 22:37:21.347502 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"thread_start_us":1137,"threads_started":1}
I20260809 22:37:21.347887 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.339025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.28.243.65:36309)
0809 22:37:21.344527 (+ 5502us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.344537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.344590 (+ 53us) server_negotiation.cc:407] Connection header received
0809 22:37:21.344662 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.344669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.344851 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.345071 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.347001 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.347012 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.347025 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.347047 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.347115 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.347139 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.347366 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.347436 (+ 70us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.347533 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5318,"thread_start_us":5159,"threads_started":1}
I20260809 22:37:21.350615 2630 tablet_service.cc:1467] Processing CreateTablet for tablet 25e05d3b400f4ae1995588db23aaa2a2 (DEFAULT_TABLE table=test-table [id=68616962b10646d282953a8585c661e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:21.350615 2492 tablet_service.cc:1467] Processing CreateTablet for tablet 25e05d3b400f4ae1995588db23aaa2a2 (DEFAULT_TABLE table=test-table [id=68616962b10646d282953a8585c661e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:21.351112 2350 tablet_service.cc:1467] Processing CreateTablet for tablet 25e05d3b400f4ae1995588db23aaa2a2 (DEFAULT_TABLE table=test-table [id=68616962b10646d282953a8585c661e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:21.352183 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e05d3b400f4ae1995588db23aaa2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:21.352632 2492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e05d3b400f4ae1995588db23aaa2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:21.352676 2350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e05d3b400f4ae1995588db23aaa2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:21.366004 2716 tablet_bootstrap.cc:492] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: Bootstrap starting.
I20260809 22:37:21.368299 2717 tablet_bootstrap.cc:492] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Bootstrap starting.
I20260809 22:37:21.370023 2718 tablet_bootstrap.cc:492] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Bootstrap starting.
I20260809 22:37:21.371899 2716 tablet_bootstrap.cc:654] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:21.372277 2717 tablet_bootstrap.cc:654] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:21.373121 2718 tablet_bootstrap.cc:654] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:21.373474 2716 log.cc:824] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:21.373615 2717 log.cc:824] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:21.374197 2718 log.cc:824] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:21.376978 2717 tablet_bootstrap.cc:492] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: No bootstrap required, opened a new log
I20260809 22:37:21.376993 2716 tablet_bootstrap.cc:492] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: No bootstrap required, opened a new log
I20260809 22:37:21.377205 2718 tablet_bootstrap.cc:492] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: No bootstrap required, opened a new log
I20260809 22:37:21.377274 2717 ts_tablet_manager.cc:1397] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260809 22:37:21.377274 2716 ts_tablet_manager.cc:1397] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260809 22:37:21.377504 2718 ts_tablet_manager.cc:1397] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 22:37:21.386193 2718 raft_consensus.cc:348] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.386685 2718 raft_consensus.cc:374] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:21.386807 2718 raft_consensus.cc:729] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41670ae2de944059515013c58688770, State: Initialized, Role: FOLLOWER
I20260809 22:37:21.387539 2718 consensus_queue.cc:260] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.388890 2698 heartbeater.cc:503] Master 127.28.243.126:39713 was elected leader, sending a full tablet report...
I20260809 22:37:21.389300 2716 raft_consensus.cc:348] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.389300 2717 raft_consensus.cc:348] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.389801 2717 raft_consensus.cc:374] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:21.389880 2716 raft_consensus.cc:374] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:21.389973 2717 raft_consensus.cc:729] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4369e9938fb42fa9c2fb4a2a2d8656d, State: Initialized, Role: FOLLOWER
I20260809 22:37:21.390056 2716 raft_consensus.cc:729] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437f8a900d5d46e0ae02c6187542c111, State: Initialized, Role: FOLLOWER
I20260809 22:37:21.390851 2717 consensus_queue.cc:260] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.391741 2716 consensus_queue.cc:260] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.392143 2718 ts_tablet_manager.cc:1428] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Time spent starting tablet: real 0.015s user 0.007s sys 0.004s
I20260809 22:37:21.393568 2561 heartbeater.cc:503] Master 127.28.243.126:39713 was elected leader, sending a full tablet report...
I20260809 22:37:21.394454 2717 ts_tablet_manager.cc:1428] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260809 22:37:21.396533 2716 ts_tablet_manager.cc:1428] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: Time spent starting tablet: real 0.019s user 0.014s sys 0.001s
I20260809 22:37:21.485981 2723 raft_consensus.cc:482] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:21.486406 2723 raft_consensus.cc:504] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.488364 2723 leader_election.cc:288] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 437f8a900d5d46e0ae02c6187542c111 (127.28.243.65:36309), f4369e9938fb42fa9c2fb4a2a2d8656d (127.28.243.66:44209)
I20260809 22:37:21.491796 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.488775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.28.243.67:54081)
0809 22:37:21.489055 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.489081 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:21.489267 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:21.489881 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:21.489887 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:21.489912 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:21.490693 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:21.490698 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:21.490718 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:21.490779 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:21.491334 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:21.491339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:21.491346 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 22:37:21.491506 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 22:37:21.491562 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 22:37:21.491837 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.488889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:54081 (local address 127.28.243.65:36309)
0809 22:37:21.489093 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.489101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.489198 (+ 97us) server_negotiation.cc:407] Connection header received
0809 22:37:21.489471 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.489480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.489568 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.489697 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.491040 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.491094 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.491102 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.491125 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.491170 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.491188 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.491349 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.491493 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.491545 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:37:21.492532 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.489413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:48561 (local address 127.28.243.66:44209)
0809 22:37:21.489713 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.489721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.490133 (+ 412us) server_negotiation.cc:407] Connection header received
0809 22:37:21.490282 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.490288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.490350 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.490469 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.491676 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.491683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.491688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.491708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.491746 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.491785 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.491915 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.492281 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.492343 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260809 22:37:21.492802 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.489228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:44209 (local address 127.28.243.67:48561)
0809 22:37:21.489961 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.489984 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:21.490108 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:21.490547 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:21.490551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:21.490623 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:21.491457 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:37:21.491460 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:21.491477 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:37:21.491521 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:21.492110 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:21.492115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:21.492120 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 22:37:21.492272 (+ 152us) client_negotiation.cc:240] Negotiation successful
0809 22:37:21.492331 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":406,"threads_started":1}
I20260809 22:37:21.493193 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e05d3b400f4ae1995588db23aaa2a2" candidate_uuid: "a41670ae2de944059515013c58688770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437f8a900d5d46e0ae02c6187542c111" is_pre_election: true
I20260809 22:37:21.493804 2373 raft_consensus.cc:2454] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41670ae2de944059515013c58688770 in term 0.
I20260809 22:37:21.493804 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e05d3b400f4ae1995588db23aaa2a2" candidate_uuid: "a41670ae2de944059515013c58688770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" is_pre_election: true
I20260809 22:37:21.494401 2514 raft_consensus.cc:2454] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41670ae2de944059515013c58688770 in term 0.
I20260809 22:37:21.494498 2582 leader_election.cc:302] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437f8a900d5d46e0ae02c6187542c111, a41670ae2de944059515013c58688770; no voters:
I20260809 22:37:21.495054 2723 raft_consensus.cc:2781] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:21.495326 2723 raft_consensus.cc:482] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:21.495461 2723 raft_consensus.cc:3037] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:21.498044 2723 raft_consensus.cc:504] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.498646 2723 leader_election.cc:288] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [CANDIDATE]: Term 1 election: Requested vote from peers 437f8a900d5d46e0ae02c6187542c111 (127.28.243.65:36309), f4369e9938fb42fa9c2fb4a2a2d8656d (127.28.243.66:44209)
I20260809 22:37:21.499218 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e05d3b400f4ae1995588db23aaa2a2" candidate_uuid: "a41670ae2de944059515013c58688770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437f8a900d5d46e0ae02c6187542c111"
I20260809 22:37:21.499272 2514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e05d3b400f4ae1995588db23aaa2a2" candidate_uuid: "a41670ae2de944059515013c58688770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d"
I20260809 22:37:21.499466 2373 raft_consensus.cc:3037] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:21.499492 2514 raft_consensus.cc:3037] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:21.502902 2514 raft_consensus.cc:2454] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41670ae2de944059515013c58688770 in term 1.
I20260809 22:37:21.502902 2373 raft_consensus.cc:2454] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41670ae2de944059515013c58688770 in term 1.
I20260809 22:37:21.503386 2582 leader_election.cc:302] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41670ae2de944059515013c58688770, f4369e9938fb42fa9c2fb4a2a2d8656d; no voters:
I20260809 22:37:21.503681 2723 raft_consensus.cc:2781] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:21.504218 2723 raft_consensus.cc:686] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 1 LEADER]: Becoming Leader. State: Replica: a41670ae2de944059515013c58688770, State: Running, Role: LEADER
I20260809 22:37:21.504879 2723 consensus_queue.cc:237] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } }
I20260809 22:37:21.510248 2219 catalog_manager.cc:5495] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41670ae2de944059515013c58688770 (127.28.243.67). New cstate: current_term: 1 leader_uuid: "a41670ae2de944059515013c58688770" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41670ae2de944059515013c58688770" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46687 } health_report { overall_health: HEALTHY } } }
W20260809 22:37:21.529047 2699 tablet.cc:2369] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:37:21.536936 2562 tablet.cc:2369] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:21.544667 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.542655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.28.243.67:46687)
0809 22:37:21.542853 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.542858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.542876 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:21.543037 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.543042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.543132 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.543224 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.544089 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.544094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.544097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.544111 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.544146 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.544158 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.544260 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.544356 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.544404 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 22:37:21.552403 2422 tablet.cc:2369] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:21.553370 2373 raft_consensus.cc:1261] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 1 FOLLOWER]: Refusing update from remote peer a41670ae2de944059515013c58688770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:21.553369 2514 raft_consensus.cc:1261] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 1 FOLLOWER]: Refusing update from remote peer a41670ae2de944059515013c58688770: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:21.554447 2727 consensus_queue.cc:1035] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437f8a900d5d46e0ae02c6187542c111" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:21.555002 2723 consensus_queue.cc:1035] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4369e9938fb42fa9c2fb4a2a2d8656d" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:21.570132 2732 mvcc.cc:205] Tried to move back new op lower bound from 7316746410191790080 to 7316746410008862720. Current Snapshot: MvccSnapshot[applied={T|T < 7316746410191790080}]
I20260809 22:37:21.570998 2730 mvcc.cc:205] Tried to move back new op lower bound from 7316746410191790080 to 7316746410008862720. Current Snapshot: MvccSnapshot[applied={T|T < 7316746410191790080}]
I20260809 22:37:21.583081 2219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:21.583386 2219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:21.587484 2219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4f49abef7ae4b77af46ace5e5ebf958: Sending DeleteTablet for 3 replicas of tablet 25e05d3b400f4ae1995588db23aaa2a2
I20260809 22:37:21.589013 2630 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e05d3b400f4ae1995588db23aaa2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:21 UTC) from {username='slave'} at 127.0.0.1:54510
I20260809 22:37:21.589241 2350 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e05d3b400f4ae1995588db23aaa2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:21 UTC) from {username='slave'} at 127.0.0.1:35536
I20260809 22:37:21.589407 2492 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e05d3b400f4ae1995588db23aaa2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:21 UTC) from {username='slave'} at 127.0.0.1:43986
I20260809 22:37:21.591344 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2283
I20260809 22:37:21.602298 2741 tablet_replica.cc:330] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: stopping tablet replica
I20260809 22:37:21.602937 2742 tablet_replica.cc:330] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111: stopping tablet replica
I20260809 22:37:21.603447 2743 tablet_replica.cc:330] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: stopping tablet replica
I20260809 22:37:21.603930 2743 raft_consensus.cc:2229] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:21.603920 2741 raft_consensus.cc:2229] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:21.604350 2742 raft_consensus.cc:2229] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:21.604609 2743 raft_consensus.cc:2258] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:21.605042 2742 raft_consensus.cc:2258] T 25e05d3b400f4ae1995588db23aaa2a2 P 437f8a900d5d46e0ae02c6187542c111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:21.605270 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.593273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.28.243.65:36309)
0809 22:37:21.593550 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.593561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.597772 (+ 4211us) server_negotiation.cc:407] Connection header received
0809 22:37:21.598169 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.598180 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.598364 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.598514 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.604136 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.604146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.604154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.604185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.604231 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.604249 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.604395 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.604800 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.604880 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 22:37:21.605733 2741 raft_consensus.cc:2258] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:21.606647 2415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:21.608533 2741 ts_tablet_manager.cc:1905] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:21.608599 2743 ts_tablet_manager.cc:1905] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:21.617903 2741 ts_tablet_manager.cc:1918] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:21.618279 2741 log.cc:1196] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/25e05d3b400f4ae1995588db23aaa2a2
I20260809 22:37:21.619073 2741 ts_tablet_manager.cc:1939] T 25e05d3b400f4ae1995588db23aaa2a2 P a41670ae2de944059515013c58688770: Deleting consensus metadata
I20260809 22:37:21.619860 2743 ts_tablet_manager.cc:1918] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:21.620144 2743 log.cc:1196] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/25e05d3b400f4ae1995588db23aaa2a2
I20260809 22:37:21.620973 2743 ts_tablet_manager.cc:1939] T 25e05d3b400f4ae1995588db23aaa2a2 P f4369e9938fb42fa9c2fb4a2a2d8656d: Deleting consensus metadata
I20260809 22:37:21.621568 2204 catalog_manager.cc:4841] TS a41670ae2de944059515013c58688770 (127.28.243.67:46687): tablet 25e05d3b400f4ae1995588db23aaa2a2 (table test-table [id=68616962b10646d282953a8585c661e3]) successfully deleted
I20260809 22:37:21.623350 2207 catalog_manager.cc:4841] TS f4369e9938fb42fa9c2fb4a2a2d8656d (127.28.243.66:44209): tablet 25e05d3b400f4ae1995588db23aaa2a2 (table test-table [id=68616962b10646d282953a8585c661e3]) successfully deleted
I20260809 22:37:21.756239 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2283
W20260809 22:37:21.772899 2207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.65:36309 (error 108)
I20260809 22:37:21.773684 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2425
I20260809 22:37:21.775535 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.775028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35560)
0809 22:37:21.775190 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.775390 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 22:37:21.775832 2207 catalog_manager.cc:4584] TS 437f8a900d5d46e0ae02c6187542c111 (127.28.243.65:36309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25e05d3b400f4ae1995588db23aaa2a2: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
I20260809 22:37:21.777472 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.774640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.28.243.66:44209)
0809 22:37:21.774842 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.774849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.775345 (+ 496us) server_negotiation.cc:407] Connection header received
0809 22:37:21.775422 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.775428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.775540 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.775638 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.776776 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.776780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.776784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.776800 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.776835 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.776847 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.776965 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.777114 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.777167 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 22:37:21.778266 2555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:21.829777 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.829157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35572)
0809 22:37:21.829451 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.829565 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260809 22:37:21.831238 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.830780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35576)
0809 22:37:21.830977 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.831090 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 22:37:21.875630 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2425
I20260809 22:37:21.892335 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.891674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35590)
0809 22:37:21.891975 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.892130 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260809 22:37:21.893821 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.893305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35598)
0809 22:37:21.893506 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.893654 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 22:37:21.897116 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2564
I20260809 22:37:21.901321 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.897999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.28.243.67:46687)
0809 22:37:21.898198 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:21.898203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:21.898221 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:21.898590 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:21.898599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:21.898711 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:21.898870 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:21.900099 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:21.900108 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:21.900113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:21.900143 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:21.900202 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:21.900231 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:21.900384 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:21.900848 (+ 464us) server_negotiation.cc:299] Negotiation successful
0809 22:37:21.900936 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 22:37:21.902145 2693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:21.974785 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.974254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35604)
0809 22:37:21.974514 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.974619 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260809 22:37:21.975891 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:21.975499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36309 (local address 127.0.0.1:35616)
0809 22:37:21.975703 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:21.975772 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:36309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260809 22:37:22.027275 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2564
I20260809 22:37:22.049178 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2188
I20260809 22:37:22.050556 2251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:22.164448 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2188
2026-08-09T22:37:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3045 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 22:37:22.211310 29645 mini_oidc.cc:152] Starting JWKS server
I20260809 22:37:22.214831 29645 webserver.cc:460] Webserver started at https://127.0.0.1:36465/ using document root <none> and password file <none>
I20260809 22:37:22.215292 29645 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 22:37:22.216182 29645 webserver.cc:460] Webserver started at http://127.0.0.1:41357/ using document root <none> and password file <none>
2026-08-09T22:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:22Z Disabled control of system clock
I20260809 22:37:22.253080 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33365
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33365
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36465/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/testchainca.cert with env {}
W20260809 22:37:22.607153 2756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:22.607556 2756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:22.607724 2756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:22.615392 2756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:22.615571 2756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:22.615654 2756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:22.615761 2756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:22.615851 2756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 22:37:22.615944 2756 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 22:37:22.616021 2756 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36465/jwks.json
I20260809 22:37:22.626595 2756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36465/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:22.628021 2756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:22.629457 2756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:22.642077 2762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:22.642289 2765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:22.642264 2763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:22.643739 2756 server_base.cc:1034] running on GCE node
I20260809 22:37:22.644680 2756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:22.646239 2756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:22.647521 2756 hybrid_clock.cc:648] HybridClock initialized: now 1786315042647473 us; error 41 us; skew 500 ppm
I20260809 22:37:22.647877 2756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:22.654754 2756 webserver.cc:460] Webserver started at http://127.28.243.126:40953/ using document root <none> and password file <none>
I20260809 22:37:22.655545 2756 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:22.655659 2756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:22.655972 2756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:22.659372 2756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "ea9b8895c47d4ef585755df43c7f4506"
format_stamp: "Formatted at 2026-08-09 22:37:22 on dist-test-slave-rgb1"
server_key: "668c2dbb3b810aa70295fd307eda1444"
server_key_iv: "8b8d7ce7ba9805db78e65d055148f89b"
server_key_version: "encryptionkey@0"
I20260809 22:37:22.660199 2756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "ea9b8895c47d4ef585755df43c7f4506"
format_stamp: "Formatted at 2026-08-09 22:37:22 on dist-test-slave-rgb1"
server_key: "668c2dbb3b810aa70295fd307eda1444"
server_key_iv: "8b8d7ce7ba9805db78e65d055148f89b"
server_key_version: "encryptionkey@0"
I20260809 22:37:22.666119 2756 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260809 22:37:22.670994 2772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:22.672716 2756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:22.672937 2756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "ea9b8895c47d4ef585755df43c7f4506"
format_stamp: "Formatted at 2026-08-09 22:37:22 on dist-test-slave-rgb1"
server_key: "668c2dbb3b810aa70295fd307eda1444"
server_key_iv: "8b8d7ce7ba9805db78e65d055148f89b"
server_key_version: "encryptionkey@0"
I20260809 22:37:22.673252 2756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:22.690730 2756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:22.691687 2756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:22.692028 2756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:22.711728 2756 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33365
I20260809 22:37:22.711750 2824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33365 every 8 connection(s)
I20260809 22:37:22.713241 2756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:22.717864 2825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:22.719313 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2756
I20260809 22:37:22.719592 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:22.720319 29645 external_mini_cluster.cc:1428] Setting key 4ca6079111ab208d28bfd71a54f03e6e
I20260809 22:37:22.732987 2825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506: Bootstrap starting.
I20260809 22:37:22.737323 2825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:22.738173 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:22.722512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.28.243.126:33365)
0809 22:37:22.723843 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:22.723860 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:22.723925 (+ 65us) server_negotiation.cc:407] Connection header received
0809 22:37:22.726584 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:22.726616 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:22.727235 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:22.727714 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:22.728356 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:22.730798 (+ 2442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:22.732318 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:22.733404 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:22.733611 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:22.735020 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:22.735032 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:22.735051 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:22.735829 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:22.735963 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:22.735994 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:22.736107 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:22.736404 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 22:37:22.736792 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":443,"threads_started":1}
I20260809 22:37:22.738705 2825 log.cc:824] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:22.741901 2825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506: No bootstrap required, opened a new log
I20260809 22:37:22.742461 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:37:22.750257 2825 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9b8895c47d4ef585755df43c7f4506" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33365 } }
I20260809 22:37:22.750885 2825 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:22.751039 2825 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9b8895c47d4ef585755df43c7f4506, State: Initialized, Role: FOLLOWER
I20260809 22:37:22.751588 2825 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9b8895c47d4ef585755df43c7f4506" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33365 } }
I20260809 22:37:22.751804 2825 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:22.751911 2825 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:22.752110 2825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:22.754899 2825 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9b8895c47d4ef585755df43c7f4506" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33365 } }
I20260809 22:37:22.755430 2825 leader_election.cc:302] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9b8895c47d4ef585755df43c7f4506; no voters:
I20260809 22:37:22.756071 2825 leader_election.cc:288] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:22.756269 2830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:22.757210 2830 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [term 1 LEADER]: Becoming Leader. State: Replica: ea9b8895c47d4ef585755df43c7f4506, State: Running, Role: LEADER
I20260809 22:37:22.758163 2830 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9b8895c47d4ef585755df43c7f4506" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33365 } }
I20260809 22:37:22.758603 2825 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:22.761531 2833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9b8895c47d4ef585755df43c7f4506. Latest consensus state: current_term: 1 leader_uuid: "ea9b8895c47d4ef585755df43c7f4506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9b8895c47d4ef585755df43c7f4506" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33365 } } }
I20260809 22:37:22.762037 2833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:22.762972 2831 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9b8895c47d4ef585755df43c7f4506" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9b8895c47d4ef585755df43c7f4506" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33365 } } }
I20260809 22:37:22.763410 2831 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:22.763803 2840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:22.771312 2840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:22.781898 2840 catalog_manager.cc:1347] Generated new cluster ID: 27f4e8258fd745d3a4cf0d47593cdaba
I20260809 22:37:22.782042 2840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:22.801964 2840 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:22.803364 2840 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:22.809757 2840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9b8895c47d4ef585755df43c7f4506: Generated new TSK 0
I20260809 22:37:22.810890 2840 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 22:37:22.864848 2856 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 22:37:22.865052 2856 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 22:37:22.867710 2828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:22.842186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.28.243.126:33365)
0809 22:37:22.842480 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:22.842488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:22.842889 (+ 401us) server_negotiation.cc:407] Connection header received
0809 22:37:22.843297 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:22.843301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:22.843387 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:22.843516 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0809 22:37:22.844212 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:22.845198 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:22.847452 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:22.848076 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:22.848206 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:22.849077 (+ 871us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 22:37:22.867136 (+ 18059us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 22:37:22.867469 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 22:37:22.867868 2828 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56444: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 22:37:22.870059 2855 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:22.842072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33365 (local address 127.0.0.1:56444)
0809 22:37:22.842767 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:22.842789 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:22.842991 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:22.843570 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:22.843575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:22.843589 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0809 22:37:22.844011 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:22.844024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:22.845358 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:22.845364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:22.847274 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:22.847289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:22.848296 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:22.848302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:22.848520 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:22.848651 (+ 131us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 22:37:22.867498 (+ 18847us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 22:37:22.867878 (+ 380us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.126:33365: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":342,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.243.126:33365: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36465/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 22:37:22.872316 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2756
I20260809 22:37:22.873764 2820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:22.972958 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2756
2026-08-09T22:37:23Z chronyd exiting
I20260809 22:37:23.027729 29645 test_util.cc:175] -----------------------------------------------
I20260809 22:37:23.027917 29645 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786314994965430-29645-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (822 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 22:37:23.033335 29645 mini_oidc.cc:152] Starting JWKS server
I20260809 22:37:23.035317 29645 webserver.cc:460] Webserver started at https://127.0.0.1:37041/ using document root <none> and password file <none>
I20260809 22:37:23.035590 29645 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 22:37:23.036335 29645 webserver.cc:460] Webserver started at http://127.0.0.1:36075/ using document root <none> and password file <none>
2026-08-09T22:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:23Z Disabled control of system clock
I20260809 22:37:23.071807 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:37465
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:34605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:37465
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37041/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/testchainca.cert with env {}
W20260809 22:37:23.424167 2864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:23.424695 2864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:23.424872 2864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:23.432847 2864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:23.433027 2864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:23.433107 2864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:23.433223 2864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:23.433302 2864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 22:37:23.433393 2864 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 22:37:23.433471 2864 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37041/jwks.json
I20260809 22:37:23.444096 2864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:34605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:37465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:37465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37041/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:23.445581 2864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:23.446959 2864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:23.460294 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:23.460429 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:23.460580 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:23.461742 2864 server_base.cc:1034] running on GCE node
I20260809 22:37:23.462692 2864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:23.464319 2864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:23.465658 2864 hybrid_clock.cc:648] HybridClock initialized: now 1786315043465591 us; error 99 us; skew 500 ppm
I20260809 22:37:23.466156 2864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:23.468926 2864 webserver.cc:460] Webserver started at http://127.28.243.126:36827/ using document root <none> and password file <none>
I20260809 22:37:23.469723 2864 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:23.469847 2864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:23.470284 2864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:23.474341 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "027fefb3283c488592959593e946ea3f"
format_stamp: "Formatted at 2026-08-09 22:37:23 on dist-test-slave-rgb1"
server_key: "2ca3b0518b38b3a5f8764accbd0ca005"
server_key_iv: "8f1e0a303ae7eb8b80c10edbc3d3bedc"
server_key_version: "encryptionkey@0"
I20260809 22:37:23.475528 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "027fefb3283c488592959593e946ea3f"
format_stamp: "Formatted at 2026-08-09 22:37:23 on dist-test-slave-rgb1"
server_key: "2ca3b0518b38b3a5f8764accbd0ca005"
server_key_iv: "8f1e0a303ae7eb8b80c10edbc3d3bedc"
server_key_version: "encryptionkey@0"
I20260809 22:37:23.481418 2864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 22:37:23.486070 2878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:23.487738 2864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260809 22:37:23.487972 2864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "027fefb3283c488592959593e946ea3f"
format_stamp: "Formatted at 2026-08-09 22:37:23 on dist-test-slave-rgb1"
server_key: "2ca3b0518b38b3a5f8764accbd0ca005"
server_key_iv: "8f1e0a303ae7eb8b80c10edbc3d3bedc"
server_key_version: "encryptionkey@0"
I20260809 22:37:23.488268 2864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:23.510349 2864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:23.511350 2864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:23.511722 2864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:23.533682 2864 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:37465
I20260809 22:37:23.533725 2930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:37465 every 8 connection(s)
I20260809 22:37:23.535277 2864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:23.537528 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2864
I20260809 22:37:23.537756 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:23.538497 29645 external_mini_cluster.cc:1428] Setting key 06899a7ba112998fd25c60e697268a2f
I20260809 22:37:23.539539 2931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:23.555299 2931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f: Bootstrap starting.
I20260809 22:37:23.555706 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:23.540419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.28.243.126:37465)
0809 22:37:23.541776 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:23.541788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:23.541843 (+ 55us) server_negotiation.cc:407] Connection header received
0809 22:37:23.544755 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:23.544787 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:23.545327 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:23.545730 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:23.546480 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:23.548093 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:23.550160 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:23.551013 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:23.551189 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:23.552814 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:23.552855 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:23.552880 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:23.553622 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:23.553717 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:23.553743 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:23.553852 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:23.554218 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 22:37:23.554622 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":452,"threads_started":1}
I20260809 22:37:23.559109 2931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:23.560447 2931 log.cc:824] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:23.560730 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:37:23.564271 2931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f: No bootstrap required, opened a new log
I20260809 22:37:23.573654 2931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027fefb3283c488592959593e946ea3f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 37465 } }
I20260809 22:37:23.574091 2931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:23.574203 2931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027fefb3283c488592959593e946ea3f, State: Initialized, Role: FOLLOWER
I20260809 22:37:23.574944 2931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027fefb3283c488592959593e946ea3f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 37465 } }
I20260809 22:37:23.575248 2931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:23.575419 2931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:23.575619 2931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:23.578074 2931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027fefb3283c488592959593e946ea3f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 37465 } }
I20260809 22:37:23.578562 2931 leader_election.cc:302] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 027fefb3283c488592959593e946ea3f; no voters:
I20260809 22:37:23.579191 2931 leader_election.cc:288] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:23.579334 2937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:23.580737 2937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [term 1 LEADER]: Becoming Leader. State: Replica: 027fefb3283c488592959593e946ea3f, State: Running, Role: LEADER
I20260809 22:37:23.581266 2937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027fefb3283c488592959593e946ea3f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 37465 } }
I20260809 22:37:23.581629 2931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:23.584203 2939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 027fefb3283c488592959593e946ea3f. Latest consensus state: current_term: 1 leader_uuid: "027fefb3283c488592959593e946ea3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027fefb3283c488592959593e946ea3f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 37465 } } }
I20260809 22:37:23.584620 2939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:23.584288 2938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "027fefb3283c488592959593e946ea3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "027fefb3283c488592959593e946ea3f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 37465 } } }
I20260809 22:37:23.584753 2938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:23.586277 2944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:23.593243 2944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:23.603513 2944 catalog_manager.cc:1347] Generated new cluster ID: 74f1d1e99eb2452681fab9afb17d3525
I20260809 22:37:23.603718 2944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:23.635937 2944 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:23.638195 2944 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:23.655135 2944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 027fefb3283c488592959593e946ea3f: Generated new TSK 0
I20260809 22:37:23.656158 2944 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 22:37:23.708649 2965 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 22:37:23.708801 2965 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 22:37:23.710300 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:23.687249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.28.243.126:37465)
0809 22:37:23.687475 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:23.687483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:23.687923 (+ 440us) server_negotiation.cc:407] Connection header received
0809 22:37:23.688173 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:23.688181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:23.688276 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:23.688467 (+ 191us) server_negotiation.cc:226] Negotiated authn=JWT
0809 22:37:23.689061 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:23.690231 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:23.692098 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:23.692998 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:23.693140 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:23.693890 (+ 750us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 22:37:23.709479 (+ 15589us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 22:37:23.709943 (+ 464us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 22:37:23.710413 2964 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:23.687071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:37465 (local address 127.0.0.1:33788)
0809 22:37:23.687785 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:23.687805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:23.687991 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:23.688493 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:23.688497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:23.688507 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 22:37:23.688823 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:23.688831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:23.690427 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:23.690433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:23.691904 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:23.691918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:23.693202 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:23.693207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:23.693463 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:23.693506 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 22:37:23.709809 (+ 16303us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 22:37:23.710039 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.126:37465: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":433,"threads_started":1}
W20260809 22:37:23.710445 2935 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33788: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37041/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260809 22:37:23.711975 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2864
I20260809 22:37:23.713402 2926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:23.813167 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2864
2026-08-09T22:37:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (840 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 22:37:23.872900 29645 mini_oidc.cc:152] Starting JWKS server
I20260809 22:37:23.874893 29645 webserver.cc:460] Webserver started at https://127.0.0.1:33723/ using document root <none> and password file <none>
I20260809 22:37:23.875180 29645 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 22:37:23.875958 29645 webserver.cc:460] Webserver started at http://127.0.0.1:35121/ using document root <none> and password file <none>
2026-08-09T22:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:23Z Disabled control of system clock
I20260809 22:37:23.912125 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33571
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33571
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33723/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 22:37:24.262809 2972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:24.263213 2972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:24.263381 2972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:24.271385 2972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:24.271579 2972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:24.271675 2972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:24.271792 2972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:24.271870 2972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 22:37:24.271965 2972 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 22:37:24.272037 2972 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33723/jwks.json
I20260809 22:37:24.283478 2972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33723/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:24.285059 2972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:24.287230 2972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:24.300798 2977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:24.300858 2980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:24.300760 2978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:24.301987 2972 server_base.cc:1034] running on GCE node
I20260809 22:37:24.302887 2972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:24.304390 2972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:24.305655 2972 hybrid_clock.cc:648] HybridClock initialized: now 1786315044305606 us; error 59 us; skew 500 ppm
I20260809 22:37:24.306156 2972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:24.308904 2972 webserver.cc:460] Webserver started at http://127.28.243.126:44587/ using document root <none> and password file <none>
I20260809 22:37:24.309708 2972 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:24.309811 2972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:24.310201 2972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:24.313383 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "142f6301ba50469cbee823e087fe17bb"
format_stamp: "Formatted at 2026-08-09 22:37:24 on dist-test-slave-rgb1"
server_key: "21c40593377d64cad600f9bdb6bdb216"
server_key_iv: "d7e4f9757edc1a18a6e7031159e18e5a"
server_key_version: "encryptionkey@0"
I20260809 22:37:24.314239 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "142f6301ba50469cbee823e087fe17bb"
format_stamp: "Formatted at 2026-08-09 22:37:24 on dist-test-slave-rgb1"
server_key: "21c40593377d64cad600f9bdb6bdb216"
server_key_iv: "d7e4f9757edc1a18a6e7031159e18e5a"
server_key_version: "encryptionkey@0"
I20260809 22:37:24.320065 2972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 22:37:24.324092 2987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:24.325906 2972 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:24.326162 2972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "142f6301ba50469cbee823e087fe17bb"
format_stamp: "Formatted at 2026-08-09 22:37:24 on dist-test-slave-rgb1"
server_key: "21c40593377d64cad600f9bdb6bdb216"
server_key_iv: "d7e4f9757edc1a18a6e7031159e18e5a"
server_key_version: "encryptionkey@0"
I20260809 22:37:24.326404 2972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:24.353951 2972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:24.354889 2972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:24.355237 2972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:24.376775 2972 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33571
I20260809 22:37:24.376744 3040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33571 every 8 connection(s)
I20260809 22:37:24.378293 2972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:24.383167 3041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:24.387290 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 2972
I20260809 22:37:24.387565 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:24.388229 29645 external_mini_cluster.cc:1428] Setting key 0bee2fb91d574ee0fc2ad3979c97983c
I20260809 22:37:24.397783 3041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb: Bootstrap starting.
I20260809 22:37:24.402094 3041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:24.403828 3041 log.cc:824] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:24.407200 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:24.390614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.28.243.126:33571)
0809 22:37:24.391803 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:24.391821 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:24.391881 (+ 60us) server_negotiation.cc:407] Connection header received
0809 22:37:24.395555 (+ 3674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:24.395595 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:24.396328 (+ 733us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:24.396875 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:24.397512 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:24.399851 (+ 2339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:24.401457 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:24.402759 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:24.402966 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:24.404692 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:24.404714 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:24.404743 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:24.405405 (+ 662us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:24.405500 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:24.405524 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:24.405672 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:24.405924 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 22:37:24.406251 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":375,"threads_started":1}
I20260809 22:37:24.407634 3041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb: No bootstrap required, opened a new log
I20260809 22:37:24.411720 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:37:24.417138 3041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142f6301ba50469cbee823e087fe17bb" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33571 } }
I20260809 22:37:24.417587 3041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:24.417712 3041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142f6301ba50469cbee823e087fe17bb, State: Initialized, Role: FOLLOWER
I20260809 22:37:24.418361 3041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142f6301ba50469cbee823e087fe17bb" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33571 } }
I20260809 22:37:24.418644 3041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:24.418797 3041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:24.418987 3041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:24.422093 3041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142f6301ba50469cbee823e087fe17bb" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33571 } }
I20260809 22:37:24.422677 3041 leader_election.cc:302] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 142f6301ba50469cbee823e087fe17bb; no voters:
I20260809 22:37:24.423386 3041 leader_election.cc:288] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:24.423712 3046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:24.424767 3046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [term 1 LEADER]: Becoming Leader. State: Replica: 142f6301ba50469cbee823e087fe17bb, State: Running, Role: LEADER
I20260809 22:37:24.425565 3046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142f6301ba50469cbee823e087fe17bb" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33571 } }
I20260809 22:37:24.425870 3041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:24.429576 3047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "142f6301ba50469cbee823e087fe17bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142f6301ba50469cbee823e087fe17bb" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33571 } } }
I20260809 22:37:24.430078 3047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:24.433084 3048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 142f6301ba50469cbee823e087fe17bb. Latest consensus state: current_term: 1 leader_uuid: "142f6301ba50469cbee823e087fe17bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "142f6301ba50469cbee823e087fe17bb" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33571 } } }
I20260809 22:37:24.433610 3048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:24.442232 3055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:24.448122 3055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:24.458388 3055 catalog_manager.cc:1347] Generated new cluster ID: fbb02c9cd7b149c9b26ad380cca7d2df
I20260809 22:37:24.458530 3055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:24.476712 3055 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:24.478626 3055 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:24.497236 3055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 142f6301ba50469cbee823e087fe17bb: Generated new TSK 0
I20260809 22:37:24.498227 3055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:37:24.532018 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:24.522784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.28.243.126:33571)
0809 22:37:24.523029 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:24.523036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:24.523315 (+ 279us) server_negotiation.cc:407] Connection header received
0809 22:37:24.523578 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:24.523582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:24.523657 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:24.523840 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0809 22:37:24.524335 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:24.525402 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:24.527183 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:24.527738 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:24.527932 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:24.528544 (+ 612us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 22:37:24.531297 (+ 2753us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 22:37:24.531662 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 22:37:24.532178 3044 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 22:37:24.534637 3072 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:24.522603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33571 (local address 127.0.0.1:55160)
0809 22:37:24.523192 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:24.523213 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:24.523385 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:24.523854 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:24.523859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:24.523871 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0809 22:37:24.524176 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:24.524185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:24.525598 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:24.525605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:24.526999 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:24.527019 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:24.528021 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:24.528032 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:24.528190 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:24.528229 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 22:37:24.531557 (+ 3328us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 22:37:24.533993 (+ 2436us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.126:33571: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33723/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":2013,"thread_start_us":442,"threads_started":1}
I20260809 22:37:24.535871 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 2972
I20260809 22:37:24.537580 3036 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 22:37:24.550060 3073 webserver.cc:533] Webserver: sslize failed:
E20260809 22:37:24.550244 3073 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 22:37:24.655463 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 2972
2026-08-09T22:37:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (842 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T22:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:24Z Disabled control of system clock
I20260809 22:37:24.753361 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:32883
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:32883
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/testchainca.cert with env {}
W20260809 22:37:25.103183 3078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:25.103502 3078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:25.103663 3078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:25.111223 3078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:25.111358 3078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:25.111397 3078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:25.111523 3078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:25.111611 3078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 22:37:25.111712 3078 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 22:37:25.111774 3078 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 22:37:25.122331 3078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:32883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:25.123773 3078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:25.125162 3078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:25.138345 3084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:25.138319 3086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:25.138317 3083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:25.138959 3078 server_base.cc:1034] running on GCE node
I20260809 22:37:25.139947 3078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:25.141901 3078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:25.143234 3078 hybrid_clock.cc:648] HybridClock initialized: now 1786315045143178 us; error 45 us; skew 500 ppm
I20260809 22:37:25.143682 3078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:25.146749 3078 webserver.cc:460] Webserver started at http://127.28.243.126:34073/ using document root <none> and password file <none>
I20260809 22:37:25.147603 3078 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:25.147756 3078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:25.148113 3078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:25.151309 3078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "1456665222ce43fe871bf195558dcd72"
format_stamp: "Formatted at 2026-08-09 22:37:25 on dist-test-slave-rgb1"
server_key: "f4215b04b931de7132bf5b1f02475df8"
server_key_iv: "b6a594b2485473c53728e85f517b33a9"
server_key_version: "encryptionkey@0"
I20260809 22:37:25.152196 3078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "1456665222ce43fe871bf195558dcd72"
format_stamp: "Formatted at 2026-08-09 22:37:25 on dist-test-slave-rgb1"
server_key: "f4215b04b931de7132bf5b1f02475df8"
server_key_iv: "b6a594b2485473c53728e85f517b33a9"
server_key_version: "encryptionkey@0"
I20260809 22:37:25.157892 3078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 22:37:25.161688 3093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:25.163058 3078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:37:25.163363 3078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "1456665222ce43fe871bf195558dcd72"
format_stamp: "Formatted at 2026-08-09 22:37:25 on dist-test-slave-rgb1"
server_key: "f4215b04b931de7132bf5b1f02475df8"
server_key_iv: "b6a594b2485473c53728e85f517b33a9"
server_key_version: "encryptionkey@0"
I20260809 22:37:25.163611 3078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:25.179946 3078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:25.181046 3078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:25.181365 3078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:25.201462 3078 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:32883
I20260809 22:37:25.201514 3147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:32883 every 8 connection(s)
I20260809 22:37:25.202961 3078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:25.207468 3148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:25.211575 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 3078
I20260809 22:37:25.211822 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:25.212839 29645 external_mini_cluster.cc:1428] Setting key de0b712e931bf45b18957135286d77d2
I20260809 22:37:25.222532 3148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72: Bootstrap starting.
I20260809 22:37:25.225878 3148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:25.226981 3148 log.cc:824] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:25.228466 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:25.215290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.28.243.126:32883)
0809 22:37:25.216669 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:25.216683 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:25.216731 (+ 48us) server_negotiation.cc:407] Connection header received
0809 22:37:25.218609 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:25.218636 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:25.219222 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:25.219723 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:25.220448 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:25.221918 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:25.223448 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:25.224219 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:25.224468 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:25.225852 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:25.225860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:25.225879 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:37:25.226562 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:25.226664 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:37:25.226687 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:25.226836 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:25.227090 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 22:37:25.227432 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":581,"threads_started":1}
I20260809 22:37:25.230465 3148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72: No bootstrap required, opened a new log
I20260809 22:37:25.232798 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:37:25.238657 3148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456665222ce43fe871bf195558dcd72" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32883 } }
I20260809 22:37:25.239074 3148 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:25.239181 3148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1456665222ce43fe871bf195558dcd72, State: Initialized, Role: FOLLOWER
I20260809 22:37:25.239742 3148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456665222ce43fe871bf195558dcd72" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32883 } }
I20260809 22:37:25.239986 3148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:25.240145 3148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:25.240337 3148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:25.242661 3148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456665222ce43fe871bf195558dcd72" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32883 } }
I20260809 22:37:25.243204 3148 leader_election.cc:302] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1456665222ce43fe871bf195558dcd72; no voters:
I20260809 22:37:25.243834 3148 leader_election.cc:288] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:25.243988 3154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:25.244805 3154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [term 1 LEADER]: Becoming Leader. State: Replica: 1456665222ce43fe871bf195558dcd72, State: Running, Role: LEADER
I20260809 22:37:25.245414 3154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456665222ce43fe871bf195558dcd72" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32883 } }
I20260809 22:37:25.246258 3148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:25.248708 3155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1456665222ce43fe871bf195558dcd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456665222ce43fe871bf195558dcd72" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32883 } } }
I20260809 22:37:25.249138 3155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:25.249146 3156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1456665222ce43fe871bf195558dcd72. Latest consensus state: current_term: 1 leader_uuid: "1456665222ce43fe871bf195558dcd72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1456665222ce43fe871bf195558dcd72" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32883 } } }
I20260809 22:37:25.249944 3156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:25.251621 3163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:25.256982 3163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:25.267247 3163 catalog_manager.cc:1347] Generated new cluster ID: 7312884e681041d7bf1c29598cbfaf0d
I20260809 22:37:25.267388 3163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:25.284900 3163 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:25.286821 3163 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:25.297175 3163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1456665222ce43fe871bf195558dcd72: Generated new TSK 0
I20260809 22:37:25.298096 3163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:37:25.411164 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:25.321180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.28.243.126:32883)
0809 22:37:25.321386 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:25.321393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:25.321755 (+ 362us) server_negotiation.cc:407] Connection header received
0809 22:37:25.321962 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:25.321970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:25.322091 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:25.322275 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0809 22:37:25.322696 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:25.323860 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:25.325418 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:25.326198 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:25.326338 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:25.326910 (+ 572us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 22:37:25.410316 (+ 83406us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 22:37:25.410808 (+ 492us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50238: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260809 22:37:25.411342 3152 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50238: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 22:37:25.411458 3180 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:25.321005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:32883 (local address 127.0.0.1:50238)
0809 22:37:25.321639 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:25.321656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:25.321777 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:25.322290 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:25.322294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:25.322303 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 22:37:25.322547 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:25.322555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:25.324051 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:25.324056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:25.325256 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:25.325269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:25.326414 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:25.326419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:25.326569 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:25.326609 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 22:37:25.410675 (+ 84066us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 22:37:25.411003 (+ 328us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.243.126:32883: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":331,"threads_started":1}
I20260809 22:37:25.412887 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 3078
I20260809 22:37:25.414342 3143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:25.521634 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 3078
2026-08-09T22:37:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (844 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (441 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (450 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:26 dist-test-slave-rgb1 krb5kdc[3195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:26 dist-test-slave-rgb1 krb5kdc[3195](info): set up 2 sockets
Aug 09 22:37:26 dist-test-slave-rgb1 krb5kdc[3195](info): commencing operation
krb5kdc: starting...
W20260809 22:37:27.586272 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:27 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:27.857344 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40201
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:40201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:28.209209 3208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:28.209563 3208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:28.209661 3208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:28.217139 3208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:28.217258 3208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:28.217305 3208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:28.217368 3208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:28.217415 3208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:28.227473 3208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:40201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:28.228854 3208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:28.230149 3208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:28.242357 3213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:28.242425 3217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:28.242393 3214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:28.243484 3208 server_base.cc:1034] running on GCE node
I20260809 22:37:28.244289 3208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:28.245872 3208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:28.247152 3208 hybrid_clock.cc:648] HybridClock initialized: now 1786315048247100 us; error 76 us; skew 500 ppm
I20260809 22:37:28.247589 3208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:28 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:28.255877 3208 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:28.257637 3208 webserver.cc:460] Webserver started at http://127.28.243.126:41723/ using document root <none> and password file <none>
I20260809 22:37:28.258438 3208 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:28.258608 3208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:28.258965 3208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:28.262424 3208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "0ac61f0e38924d7f8ab61c41abcb104d"
format_stamp: "Formatted at 2026-08-09 22:37:28 on dist-test-slave-rgb1"
server_key: "a5b4d9fefe73f47bc1c09e00f2c1c85f"
server_key_iv: "45cef0828f49aa3c705a0aca98a156ff"
server_key_version: "encryptionkey@0"
I20260809 22:37:28.263345 3208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "0ac61f0e38924d7f8ab61c41abcb104d"
format_stamp: "Formatted at 2026-08-09 22:37:28 on dist-test-slave-rgb1"
server_key: "a5b4d9fefe73f47bc1c09e00f2c1c85f"
server_key_iv: "45cef0828f49aa3c705a0aca98a156ff"
server_key_version: "encryptionkey@0"
I20260809 22:37:28.269277 3208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:37:28.273092 3226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:28.274641 3208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 22:37:28.274860 3208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "0ac61f0e38924d7f8ab61c41abcb104d"
format_stamp: "Formatted at 2026-08-09 22:37:28 on dist-test-slave-rgb1"
server_key: "a5b4d9fefe73f47bc1c09e00f2c1c85f"
server_key_iv: "45cef0828f49aa3c705a0aca98a156ff"
server_key_version: "encryptionkey@0"
I20260809 22:37:28.275095 3208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:28.289716 3208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:28.294446 3208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:28.294761 3208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:28.314448 3208 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:40201
I20260809 22:37:28.314565 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:40201 every 8 connection(s)
I20260809 22:37:28.315894 3208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:28.319753 3280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:28.325145 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 3208
I20260809 22:37:28.325383 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:28.326072 29645 external_mini_cluster.cc:1428] Setting key 8f9ef3d4d459de51ebeab42ad8ebe275
I20260809 22:37:28.330972 3280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d: Bootstrap starting.
I20260809 22:37:28.334863 3280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:28.336035 3280 log.cc:824] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:28.339416 3280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d: No bootstrap required, opened a new log
Aug 09 22:37:28 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:28.348394 3280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40201 } }
I20260809 22:37:28.348889 3280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:28.349015 3280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac61f0e38924d7f8ab61c41abcb104d, State: Initialized, Role: FOLLOWER
I20260809 22:37:28.349723 3280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40201 } }
I20260809 22:37:28.350067 3280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:28.350230 3280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:28.350407 3280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:28.351866 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:28.328539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.28.243.126:40201)
0809 22:37:28.329574 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:28.329587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:28.329630 (+ 43us) server_negotiation.cc:407] Connection header received
0809 22:37:28.331465 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:28.331493 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:28.331896 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:28.332234 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:28.334175 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:28.335655 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:28.337365 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:28.338131 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:28.338273 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:28.343321 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:28.343330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:28.343347 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:28.343405 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:28.347064 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:28.347864 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:28.347871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:28.347877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:28.348063 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:28.348472 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:28.348476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:28.348478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:28.349262 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:28.349592 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:28.350115 (+ 523us) server_negotiation.cc:299] Negotiation successful
0809 22:37:28.350768 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":472,"threads_started":1}
I20260809 22:37:28.353084 3280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40201 } }
I20260809 22:37:28.353574 3280 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac61f0e38924d7f8ab61c41abcb104d; no voters:
I20260809 22:37:28.354188 3280 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:28.354357 3287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:28.355810 3287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [term 1 LEADER]: Becoming Leader. State: Replica: 0ac61f0e38924d7f8ab61c41abcb104d, State: Running, Role: LEADER
I20260809 22:37:28.356295 3280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:28.356297 3287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40201 } }
I20260809 22:37:28.359468 3288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40201 } } }
I20260809 22:37:28.360132 3288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:28.359728 3289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac61f0e38924d7f8ab61c41abcb104d. Latest consensus state: current_term: 1 leader_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac61f0e38924d7f8ab61c41abcb104d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40201 } } }
I20260809 22:37:28.360771 3289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:28.361824 3299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:28.367548 3299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:28.377038 3299 catalog_manager.cc:1347] Generated new cluster ID: 5a5a068f3a0a47c990cba6cc15f100fd
I20260809 22:37:28.377188 3299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:28.417177 3299 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:28.418604 3299 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:28.425673 3299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac61f0e38924d7f8ab61c41abcb104d: Generated new TSK 0
I20260809 22:37:28.426630 3299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:37:28.502151 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40201
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:28.860500 3312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:28.860900 3312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:28.861056 3312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:28.868795 3312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:28.868927 3312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:28.869134 3312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:28.880869 3312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:28.882409 3312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:28.883875 3312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:28.898916 3317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:28.899080 3318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:28.898931 3320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:28.900533 3312 server_base.cc:1034] running on GCE node
I20260809 22:37:28.901196 3312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:28.902298 3312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:28.903581 3312 hybrid_clock.cc:648] HybridClock initialized: now 1786315048903487 us; error 72 us; skew 500 ppm
I20260809 22:37:28.903892 3312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:28 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:28.907617 3312 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:37:28.909397 3312 webserver.cc:460] Webserver started at http://127.28.243.65:33555/ using document root <none> and password file <none>
I20260809 22:37:28.910241 3312 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:28.910403 3312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:28.910784 3312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:28.914323 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "f63a71b53b6f4c6bb7140cf9198d1c99"
format_stamp: "Formatted at 2026-08-09 22:37:28 on dist-test-slave-rgb1"
server_key: "788fac1271d82a686259da5a1a7d7df7"
server_key_iv: "12fa69b19e175a1160e843698dd1aecd"
server_key_version: "encryptionkey@0"
I20260809 22:37:28.915279 3312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "f63a71b53b6f4c6bb7140cf9198d1c99"
format_stamp: "Formatted at 2026-08-09 22:37:28 on dist-test-slave-rgb1"
server_key: "788fac1271d82a686259da5a1a7d7df7"
server_key_iv: "12fa69b19e175a1160e843698dd1aecd"
server_key_version: "encryptionkey@0"
I20260809 22:37:28.921190 3312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:37:28.926040 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:28.927634 3312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:37:28.927870 3312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "f63a71b53b6f4c6bb7140cf9198d1c99"
format_stamp: "Formatted at 2026-08-09 22:37:28 on dist-test-slave-rgb1"
server_key: "788fac1271d82a686259da5a1a7d7df7"
server_key_iv: "12fa69b19e175a1160e843698dd1aecd"
server_key_version: "encryptionkey@0"
I20260809 22:37:28.928143 3312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:28.947582 3312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:28.952414 3312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:28.952777 3312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:28.953954 3312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:28.955493 3312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:28.955613 3312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:28.955814 3312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:28.955899 3312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:28.996176 3312 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:41777
I20260809 22:37:28.996304 3449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:41777 every 8 connection(s)
I20260809 22:37:28.997663 3312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:29.005074 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 3312
I20260809 22:37:29.005321 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:29.006040 29645 external_mini_cluster.cc:1428] Setting key 52a586385bf200424873f070305757dd
Aug 09 22:37:29 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315048, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:29.021909 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:29.000641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:51321 (local address 127.28.243.126:40201)
0809 22:37:29.001177 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:29.001185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:29.002575 (+ 1390us) server_negotiation.cc:407] Connection header received
0809 22:37:29.004641 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:29.004650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:29.004760 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:29.004904 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:29.007129 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.008010 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.010045 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.010685 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.010871 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:29.015148 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:29.015154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:29.015157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:29.015194 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:29.018397 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:29.019312 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:29.019317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:29.019319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:29.019451 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:29.019947 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:29.019951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:29.019953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:29.020241 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:29.020413 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:29.021267 (+ 854us) server_negotiation.cc:299] Negotiation successful
0809 22:37:29.021515 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":247,"threads_started":1}
I20260809 22:37:29.022929 3454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:29.001236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40201 (local address 127.28.243.65:51321)
0809 22:37:29.002177 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:29.002229 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:29.004264 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:29.005280 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:29.005296 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:29.005995 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:29.006783 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:29.006923 (+ 140us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.008218 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.008226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:29.009825 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:29.009845 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.011056 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.011070 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:29.011344 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:29.012311 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:29.012337 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:29.014853 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:29.018602 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:29.018615 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:29.018637 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:29.019152 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:29.019576 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:29.019582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:29.019586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:29.019801 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:29.020391 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:29.020399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:29.020850 (+ 451us) client_negotiation.cc:769] Sending connection context
0809 22:37:29.021153 (+ 303us) client_negotiation.cc:240] Negotiation successful
0809 22:37:29.021741 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":419,"threads_started":1}
I20260809 22:37:29.024775 3451 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40201
I20260809 22:37:29.025134 3451 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:29.026126 3451 heartbeater.cc:511] Master 127.28.243.126:40201 requested a full tablet report, sending...
I20260809 22:37:29.028812 3243 ts_manager.cc:194] Registered new tserver with Master: f63a71b53b6f4c6bb7140cf9198d1c99 (127.28.243.65:41777)
I20260809 22:37:29.031155 3243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:51321
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:37:29.069705 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40201
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:29.432489 3457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:29.432909 3457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:29.433082 3457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:29.440816 3457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:29.440938 3457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:29.441111 3457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:29.451359 3457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:29.452783 3457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:29.454123 3457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:29.467931 3462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:29.467965 3463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:29.468006 3465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:29.469153 3457 server_base.cc:1034] running on GCE node
I20260809 22:37:29.469789 3457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:29.470903 3457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:29.472174 3457 hybrid_clock.cc:648] HybridClock initialized: now 1786315049472099 us; error 68 us; skew 500 ppm
I20260809 22:37:29.472524 3457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:29 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315049, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:29.476501 3457 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:37:29.478192 3457 webserver.cc:460] Webserver started at http://127.28.243.66:34327/ using document root <none> and password file <none>
I20260809 22:37:29.478986 3457 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:29.479154 3457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:29.479635 3457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:29.482954 3457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "0e098309af474c85a35e2604194d1545"
format_stamp: "Formatted at 2026-08-09 22:37:29 on dist-test-slave-rgb1"
server_key: "8f2b9e2fa628e25579cb237588396ff2"
server_key_iv: "13b4c72a7362a6e6a99e650071485135"
server_key_version: "encryptionkey@0"
I20260809 22:37:29.483873 3457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "0e098309af474c85a35e2604194d1545"
format_stamp: "Formatted at 2026-08-09 22:37:29 on dist-test-slave-rgb1"
server_key: "8f2b9e2fa628e25579cb237588396ff2"
server_key_iv: "13b4c72a7362a6e6a99e650071485135"
server_key_version: "encryptionkey@0"
I20260809 22:37:29.489828 3457 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:37:29.493868 3473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:29.495406 3457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:37:29.495600 3457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "0e098309af474c85a35e2604194d1545"
format_stamp: "Formatted at 2026-08-09 22:37:29 on dist-test-slave-rgb1"
server_key: "8f2b9e2fa628e25579cb237588396ff2"
server_key_iv: "13b4c72a7362a6e6a99e650071485135"
server_key_version: "encryptionkey@0"
I20260809 22:37:29.495857 3457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:29.515038 3457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:29.519766 3457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:29.520098 3457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:29.521190 3457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:29.522709 3457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:29.522815 3457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:29.522974 3457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:29.523049 3457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:29.559415 3457 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:35637
I20260809 22:37:29.559460 3592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:35637 every 8 connection(s)
I20260809 22:37:29.560963 3457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:29.567019 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 3457
I20260809 22:37:29.567209 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:29.567771 29645 external_mini_cluster.cc:1428] Setting key a501b4058c02c87f53e1095fa21345d8
Aug 09 22:37:29 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315049, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:29.587019 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:29.563793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:60177 (local address 127.28.243.126:40201)
0809 22:37:29.564393 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:29.564398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:29.565618 (+ 1220us) server_negotiation.cc:407] Connection header received
0809 22:37:29.568127 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:29.568132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:29.568211 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:29.568318 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:29.570366 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.571239 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.574026 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.574564 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.574699 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:29.579739 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:29.579755 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:29.579761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:29.579848 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:29.583446 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:29.584346 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:29.584352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:29.584356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:29.584501 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:29.585029 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:29.585035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:29.585038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:29.585341 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:29.585517 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:29.586342 (+ 825us) server_negotiation.cc:299] Negotiation successful
0809 22:37:29.586607 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":281,"threads_started":1}
I20260809 22:37:29.588109 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:29.564516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40201 (local address 127.28.243.66:60177)
0809 22:37:29.565424 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:29.565469 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:29.567723 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:29.568666 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:29.568675 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:29.569382 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:29.570129 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:29.570158 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.571453 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.571461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:29.573820 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:29.573840 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:29.574834 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:29.574846 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:29.575225 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:29.576192 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:29.576234 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:29.579421 (+ 3187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:29.583658 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:29.583668 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:29.583691 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:29.584196 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:29.584676 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:29.584681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:29.584685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:29.584898 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:29.585490 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:29.585498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:29.585933 (+ 435us) client_negotiation.cc:769] Sending connection context
0809 22:37:29.586225 (+ 292us) client_negotiation.cc:240] Negotiation successful
0809 22:37:29.586835 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":362,"threads_started":1}
I20260809 22:37:29.589867 3593 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40201
I20260809 22:37:29.590229 3593 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:29.591065 3593 heartbeater.cc:511] Master 127.28.243.126:40201 requested a full tablet report, sending...
I20260809 22:37:29.592752 3243 ts_manager.cc:194] Registered new tserver with Master: 0e098309af474c85a35e2604194d1545 (127.28.243.66:35637)
I20260809 22:37:29.593611 3243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:60177
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:37:29.628501 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40201
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:29.978550 3600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:29.978928 3600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:29.979138 3600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:29.987565 3600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:29.987746 3600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:29.987936 3600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:30.000092 3600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:30.001928 3600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:30.003575 3600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:30.018162 3608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:30.018208 3606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:30.018249 3605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:30.019466 3600 server_base.cc:1034] running on GCE node
I20260809 22:37:30.020362 3600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:30.021569 3600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:30.022847 3600 hybrid_clock.cc:648] HybridClock initialized: now 1786315050022801 us; error 60 us; skew 500 ppm
I20260809 22:37:30.023254 3600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.027535 3600 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:37:30.029399 3600 webserver.cc:460] Webserver started at http://127.28.243.67:41805/ using document root <none> and password file <none>
I20260809 22:37:30.030201 3600 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:30.030375 3600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:30.030733 3600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:30.034842 3451 heartbeater.cc:503] Master 127.28.243.126:40201 was elected leader, sending a full tablet report...
I20260809 22:37:30.035272 3600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "c28458eb5d014e6d8cc20adea0cf1171"
format_stamp: "Formatted at 2026-08-09 22:37:30 on dist-test-slave-rgb1"
server_key: "a95f5b760db53d7d9cf2b1788702ab86"
server_key_iv: "69e6b8456201c5851eef509aa1e013fe"
server_key_version: "encryptionkey@0"
I20260809 22:37:30.036319 3600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "c28458eb5d014e6d8cc20adea0cf1171"
format_stamp: "Formatted at 2026-08-09 22:37:30 on dist-test-slave-rgb1"
server_key: "a95f5b760db53d7d9cf2b1788702ab86"
server_key_iv: "69e6b8456201c5851eef509aa1e013fe"
server_key_version: "encryptionkey@0"
I20260809 22:37:30.042236 3600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 22:37:30.046118 3615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:30.047699 3600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:37:30.047919 3600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "c28458eb5d014e6d8cc20adea0cf1171"
format_stamp: "Formatted at 2026-08-09 22:37:30 on dist-test-slave-rgb1"
server_key: "a95f5b760db53d7d9cf2b1788702ab86"
server_key_iv: "69e6b8456201c5851eef509aa1e013fe"
server_key_version: "encryptionkey@0"
I20260809 22:37:30.048207 3600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:30.066126 3600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:30.071743 3600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:30.072130 3600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:30.073333 3600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:30.074936 3600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:30.075047 3600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:30.075222 3600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:30.075306 3600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:30.114705 3600 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:43865
I20260809 22:37:30.114753 3735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:43865 every 8 connection(s)
I20260809 22:37:30.116436 3600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:30.118183 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 3600
I20260809 22:37:30.118348 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:30.118922 29645 external_mini_cluster.cc:1428] Setting key 8375715c279f1757b6d89b52ad2881ac
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:30.144207 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.119732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:57627 (local address 127.28.243.126:40201)
0809 22:37:30.124933 (+ 5201us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.124939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.125676 (+ 737us) server_negotiation.cc:407] Connection header received
0809 22:37:30.127617 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.127622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.127694 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.127797 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.129804 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.130706 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.132688 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.133358 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.133529 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.137778 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:30.137783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:30.137786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:30.137822 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:30.140702 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.141640 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.141644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.141646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.141779 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.142291 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.142294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.142296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.142600 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:30.142731 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:30.143644 (+ 913us) server_negotiation.cc:299] Negotiation successful
0809 22:37:30.143833 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5082,"thread_start_us":4986,"threads_started":1}
I20260809 22:37:30.145247 3740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.120513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40201 (local address 127.28.243.67:57627)
0809 22:37:30.125480 (+ 4967us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:30.125540 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:30.127327 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:30.128158 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:30.128176 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:30.128933 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:30.129596 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:30.129617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.130922 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.130936 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:30.132440 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:30.132454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.133635 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.133645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:30.133932 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.134991 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:30.135067 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:30.137543 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:30.140851 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:30.140859 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:30.140873 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:30.141494 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:30.141881 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:30.141885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:30.141887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:30.142129 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:30.142755 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:30.142761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:30.143193 (+ 432us) client_negotiation.cc:769] Sending connection context
0809 22:37:30.143489 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 22:37:30.143952 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4595,"thread_start_us":4307,"threads_started":1}
I20260809 22:37:30.147120 3736 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40201
I20260809 22:37:30.147560 3736 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:30.148530 3736 heartbeater.cc:511] Master 127.28.243.126:40201 requested a full tablet report, sending...
I20260809 22:37:30.150481 3243 ts_manager.cc:194] Registered new tserver with Master: c28458eb5d014e6d8cc20adea0cf1171 (127.28.243.67:43865)
I20260809 22:37:30.151353 3243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:57627
I20260809 22:37:30.162505 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/test-admin.keytab.
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.199013 29645 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 22:37:30.205221 3745 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:30.202226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47160 (local address 127.28.243.65:41777)
0809 22:37:30.203072 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.203080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.203111 (+ 31us) server_negotiation.cc:407] Connection header received
0809 22:37:30.203174 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.203181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.203400 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.203568 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.204727 (+ 1159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47160: BlockingRecv error: recv got EOF from 127.0.0.1:47160 (error 108)
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":524,"threads_started":1}
W20260809 22:37:30.205317 29645 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5cc' of type FILE
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.207263 29645 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.208788 29645 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:37:30.226187 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.210520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.28.243.65:41777)
0809 22:37:30.210763 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.210770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.210995 (+ 225us) server_negotiation.cc:407] Connection header received
0809 22:37:30.211246 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.211255 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.211347 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.211478 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.212606 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.213701 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.215132 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.215729 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.215836 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.219793 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:30.219807 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:30.219823 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:30.219864 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:30.222927 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.223664 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.223671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.223674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.223805 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.224212 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.224217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.224220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.224772 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:30.225129 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:30.225504 (+ 375us) server_negotiation.cc:299] Negotiation successful
0809 22:37:30.225851 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260809 22:37:30.232856 3745 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:30.230577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.28.243.65:41777)
0809 22:37:30.230842 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.230848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.231049 (+ 201us) server_negotiation.cc:407] Connection header received
0809 22:37:30.231261 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.231265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.231352 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.231464 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.232528 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47190: BlockingRecv error: recv got EOF from 127.0.0.1:47190 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260809 22:37:30.232908 29645 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5cc' of type FILE
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.234566 29645 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.235916 29645 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:37:30.251147 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.237418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.28.243.65:41777)
0809 22:37:30.237666 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.237673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.237852 (+ 179us) server_negotiation.cc:407] Connection header received
0809 22:37:30.237936 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.237940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.238002 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.238100 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.239004 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.239761 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.241246 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.241775 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.241888 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.245727 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:30.245732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:30.245736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:30.245788 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:30.248649 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.249356 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.249360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.249363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.249496 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.249849 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.249853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.249855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.250181 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:30.250433 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:30.250618 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 22:37:30.250786 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260809 22:37:30.256521 3745 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:30.254512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.28.243.65:41777)
0809 22:37:30.254732 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.254738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.255025 (+ 287us) server_negotiation.cc:407] Connection header received
0809 22:37:30.255196 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.255200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.255291 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.255408 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.256226 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47204: BlockingRecv error: recv got EOF from 127.0.0.1:47204 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260809 22:37:30.256664 29645 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786314994965430-29645-0/krb5kdc/krb5cc' of type FILE
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.258661 29645 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:30.260167 29645 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:37:30.278332 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.262228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.28.243.65:41777)
0809 22:37:30.262520 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.262527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.262690 (+ 163us) server_negotiation.cc:407] Connection header received
0809 22:37:30.262995 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.263002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.263133 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.263235 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.264474 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.265274 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.266682 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.267243 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.267361 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.272157 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:30.272163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:30.272166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:30.272204 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:30.275146 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.276029 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.276034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.276036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.276213 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.276746 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.276750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.276752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.277134 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:30.277389 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:30.277708 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 22:37:30.277883 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 22:37:30.280184 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 3312
I20260809 22:37:30.293942 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.281160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.28.243.65:41777)
0809 22:37:30.281411 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.281417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.281662 (+ 245us) server_negotiation.cc:407] Connection header received
0809 22:37:30.281798 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.281802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.281857 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.281934 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.283005 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.283696 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.285206 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.285702 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.285837 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.288037 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:30.288056 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:30.288060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:30.288097 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:30.291050 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.291801 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.291806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.291808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.291941 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.292303 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.292306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.292308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.292646 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:30.292842 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:30.293240 (+ 398us) server_negotiation.cc:299] Negotiation successful
0809 22:37:30.293506 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:37:30.294726 3444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:30.384318 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 3312
I20260809 22:37:30.405233 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 3457
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:37:30.423125 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.406272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.28.243.66:35637)
0809 22:37:30.407027 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.407035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.407062 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:37:30.407130 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.407136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.407341 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.407552 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.408643 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.409700 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.410983 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.411571 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.411728 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.416113 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:30.416128 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:30.416143 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:30.416186 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:30.419783 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.420689 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.420698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.420701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.420861 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.421301 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.421307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.421311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.421710 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:30.422078 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:30.422470 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 22:37:30.422756 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":463,"threads_started":1}
I20260809 22:37:30.424541 3588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:30.518246 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 3457
I20260809 22:37:30.536340 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 3600
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:37:30.556989 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:30.537600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.28.243.67:43865)
0809 22:37:30.538226 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:30.538233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:30.538263 (+ 30us) server_negotiation.cc:407] Connection header received
0809 22:37:30.538322 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:30.538328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:30.538549 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:30.538696 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:30.539964 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.541170 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.542501 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:30.543132 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:30.543245 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:30.548457 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:30.548472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:30.548489 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:30.548531 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:30.552676 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.554363 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.554372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.554375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.554548 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:30.555040 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:30.555045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:30.555048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:30.555565 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:30.555863 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:30.556258 (+ 395us) server_negotiation.cc:299] Negotiation successful
0809 22:37:30.556616 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":316,"threads_started":1}
I20260809 22:37:30.558434 3731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:30.647356 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 3600
I20260809 22:37:30.667155 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 3208
I20260809 22:37:30.669158 3275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:30.779538 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 3208
2026-08-09T22:37:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4376 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3765](info): set up 2 sockets
Aug 09 22:37:30 dist-test-slave-rgb1 krb5kdc[3765](info): commencing operation
krb5kdc: starting...
W20260809 22:37:32.014467 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:32 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab.
Entry for principal oryx/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab.
I20260809 22:37:32.306239 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40309
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab
--principal=oryx/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:40309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:32.666441 3779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:32.666977 3779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:32.667176 3779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:32.676707 3779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:32.676842 3779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:32.676890 3779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:32.677011 3779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:32.677091 3779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:32.687491 3779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:40309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab
--principal=oryx/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:32.689066 3779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:32.690472 3779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:32.702919 3787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:32.702973 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:32.703019 3784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:32.703980 3779 server_base.cc:1034] running on GCE node
I20260809 22:37:32.704912 3779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:32.706437 3779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:32.707710 3779 hybrid_clock.cc:648] HybridClock initialized: now 1786315052707658 us; error 40 us; skew 500 ppm
I20260809 22:37:32.708211 3779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:32 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:32.711966 3779 init.cc:377] Logged in from keytab as oryx/127.28.243.126@KRBTEST.COM (short username oryx)
I20260809 22:37:32.714069 3779 webserver.cc:460] Webserver started at http://127.28.243.126:33837/ using document root <none> and password file <none>
I20260809 22:37:32.714880 3779 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:32.715046 3779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:32.715390 3779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:32.718951 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "09d0a9690b0d4e1d987e0e8cfab5b333"
format_stamp: "Formatted at 2026-08-09 22:37:32 on dist-test-slave-rgb1"
server_key: "90a19cae147280c207168cb6aba307bb"
server_key_iv: "b93156cf1435ee890caf2f2f680fa6d0"
server_key_version: "encryptionkey@0"
I20260809 22:37:32.719825 3779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "09d0a9690b0d4e1d987e0e8cfab5b333"
format_stamp: "Formatted at 2026-08-09 22:37:32 on dist-test-slave-rgb1"
server_key: "90a19cae147280c207168cb6aba307bb"
server_key_iv: "b93156cf1435ee890caf2f2f680fa6d0"
server_key_version: "encryptionkey@0"
I20260809 22:37:32.725703 3779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 22:37:32.729516 3794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:32.731142 3779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:37:32.731364 3779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "09d0a9690b0d4e1d987e0e8cfab5b333"
format_stamp: "Formatted at 2026-08-09 22:37:32 on dist-test-slave-rgb1"
server_key: "90a19cae147280c207168cb6aba307bb"
server_key_iv: "b93156cf1435ee890caf2f2f680fa6d0"
server_key_version: "encryptionkey@0"
I20260809 22:37:32.731590 3779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:32.754019 3779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:32.758966 3779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:32.759298 3779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:32.780046 3779 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:40309
I20260809 22:37:32.780140 3850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:40309 every 8 connection(s)
I20260809 22:37:32.781600 3779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:32.784739 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 3779
I20260809 22:37:32.784989 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:32.785837 29645 external_mini_cluster.cc:1428] Setting key ba8bb6843e58aae82d3ca69c81892d91
I20260809 22:37:32.786491 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:32.802044 3851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: Bootstrap starting.
Aug 09 22:37:32 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:32.806345 3851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:32.807852 3851 log.cc:824] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:32.811865 3851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: No bootstrap required, opened a new log
I20260809 22:37:32.813467 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:32.788131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.28.243.126:40309)
0809 22:37:32.789098 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:32.789113 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:32.789164 (+ 51us) server_negotiation.cc:407] Connection header received
0809 22:37:32.791716 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:32.791748 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:32.792336 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:32.792850 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:32.794398 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:32.796385 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:32.797847 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:32.799061 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:32.799292 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:32.803942 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:32.803954 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:32.803983 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:32.804055 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:32.808343 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:32.809470 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:32.809478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:32.809484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:32.809748 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:32.810164 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:32.810169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:32.810172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:32.810897 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:32.811239 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:32.811601 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 22:37:32.812067 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":460,"threads_started":1}
I20260809 22:37:32.820289 3851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } }
I20260809 22:37:32.820734 3851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:32.820834 3851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d0a9690b0d4e1d987e0e8cfab5b333, State: Initialized, Role: FOLLOWER
I20260809 22:37:32.821378 3851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } }
I20260809 22:37:32.821602 3851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:32.821713 3851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:32.821843 3851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:32.825064 3851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } }
I20260809 22:37:32.825721 3851 leader_election.cc:302] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09d0a9690b0d4e1d987e0e8cfab5b333; no voters:
I20260809 22:37:32.826445 3851 leader_election.cc:288] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:32.826627 3856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:32.827445 3856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 1 LEADER]: Becoming Leader. State: Replica: 09d0a9690b0d4e1d987e0e8cfab5b333, State: Running, Role: LEADER
I20260809 22:37:32.828039 3856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } }
I20260809 22:37:32.828867 3851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:32.830832 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d0a9690b0d4e1d987e0e8cfab5b333. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } }
I20260809 22:37:32.831800 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:32.831429 3857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } }
I20260809 22:37:32.832373 3857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:32.834620 3862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:32.840021 3862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:32.851109 3862 catalog_manager.cc:1347] Generated new cluster ID: 92f5287dfbb847b7942995cc5466fb95
I20260809 22:37:32.851264 3862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:32.867935 3862 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:32.869572 3862 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:32.885227 3862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: Generated new TSK 0
I20260809 22:37:32.886417 3862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab.
Entry for principal oryx/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab.
I20260809 22:37:32.953492 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40309
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab
--principal=oryx/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:33.336189 3877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:33.336529 3877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:33.336694 3877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:33.344026 3877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:33.344148 3877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:33.344267 3877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:33.354446 3877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab
--principal=oryx/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:33.356030 3877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:33.357419 3877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:33.377090 3883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:33.377013 3884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:33.377067 3886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:33.378829 3877 server_base.cc:1034] running on GCE node
I20260809 22:37:33.379700 3877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:33.380894 3877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:33.382174 3877 hybrid_clock.cc:648] HybridClock initialized: now 1786315053382124 us; error 45 us; skew 500 ppm
I20260809 22:37:33.382745 3877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:33 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315053, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:33.390163 3877 init.cc:377] Logged in from keytab as oryx/127.28.243.65@KRBTEST.COM (short username oryx)
I20260809 22:37:33.391799 3877 webserver.cc:460] Webserver started at http://127.28.243.65:38029/ using document root <none> and password file <none>
I20260809 22:37:33.392540 3877 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:33.392760 3877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:33.393106 3877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:33.397305 3877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "6ef923bb58564adc9eadd9c9e9cf1346"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "c425e0cedf18d04333ab72fd92401c54"
server_key_iv: "937f52b7d41b980842300a0fbb089f63"
server_key_version: "encryptionkey@0"
I20260809 22:37:33.398600 3877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "6ef923bb58564adc9eadd9c9e9cf1346"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "c425e0cedf18d04333ab72fd92401c54"
server_key_iv: "937f52b7d41b980842300a0fbb089f63"
server_key_version: "encryptionkey@0"
I20260809 22:37:33.406483 3877 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260809 22:37:33.411062 3895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:33.412976 3877 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:33.413216 3877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "6ef923bb58564adc9eadd9c9e9cf1346"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "c425e0cedf18d04333ab72fd92401c54"
server_key_iv: "937f52b7d41b980842300a0fbb089f63"
server_key_version: "encryptionkey@0"
I20260809 22:37:33.413491 3877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:33.427682 3877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:33.432619 3877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:33.432946 3877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:33.434096 3877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:33.435753 3877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:33.435832 3877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:33.435962 3877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:33.436030 3877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:33.476127 3877 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:32887
I20260809 22:37:33.476178 4013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:32887 every 8 connection(s)
I20260809 22:37:33.477689 3877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:33.482473 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 3877
I20260809 22:37:33.482717 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:33.483569 29645 external_mini_cluster.cc:1428] Setting key ee0fcae4f532fa69198158d7b86a367e
Aug 09 22:37:33 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315053, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:33.503147 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:33.480701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:36775 (local address 127.28.243.126:40309)
0809 22:37:33.481313 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:33.481318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:33.482211 (+ 893us) server_negotiation.cc:407] Connection header received
0809 22:37:33.484254 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:33.484260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:33.484383 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:33.484524 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:33.486871 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:33.488118 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:33.490152 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:33.490849 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:33.491021 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:33.495725 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:33.495735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:33.495744 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:33.495805 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:33.499120 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:33.500291 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:33.500297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:33.500299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:33.500455 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:33.501089 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:33.501094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:33.501096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:33.501413 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:33.501546 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:33.502632 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0809 22:37:33.502837 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":388,"threads_started":1}
I20260809 22:37:33.504720 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:33.481156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.65:36775)
0809 22:37:33.481966 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:33.482057 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:33.483959 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:33.485100 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:33.485113 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:33.485950 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:33.486651 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:33.486679 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:33.488311 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:33.488316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:33.489952 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:33.489966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:33.491175 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:33.491181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:33.491403 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:33.492320 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:33.492351 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:33.495368 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:33.499306 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:33.499319 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:33.499344 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:33.500110 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:33.500629 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:33.500637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:33.500644 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:33.500927 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:33.501558 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:33.501570 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:33.502128 (+ 558us) client_negotiation.cc:769] Sending connection context
0809 22:37:33.502465 (+ 337us) client_negotiation.cc:240] Negotiation successful
0809 22:37:33.503083 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":317,"threads_started":1}
I20260809 22:37:33.506779 4014 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40309
I20260809 22:37:33.507161 4014 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:33.508070 4014 heartbeater.cc:511] Master 127.28.243.126:40309 requested a full tablet report, sending...
I20260809 22:37:33.510833 3815 ts_manager.cc:194] Registered new tserver with Master: 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887)
WARNING: no policy specified for oryx/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:37:33.512760 3815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:36775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab.
Entry for principal oryx/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab.
I20260809 22:37:33.548893 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40309
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab
--principal=oryx/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:33.904835 4020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:33.905133 4020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:33.905303 4020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:33.912825 4020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:33.912946 4020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:33.913162 4020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:33.923530 4020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab
--principal=oryx/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:33.924976 4020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:33.926373 4020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:33.940932 4025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:33.941042 4029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:33.940901 4026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:33.941848 4020 server_base.cc:1034] running on GCE node
I20260809 22:37:33.942603 4020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:33.943818 4020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:33.945072 4020 hybrid_clock.cc:648] HybridClock initialized: now 1786315053945024 us; error 56 us; skew 500 ppm
I20260809 22:37:33.945420 4020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:33 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315053, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:33.948843 4020 init.cc:377] Logged in from keytab as oryx/127.28.243.66@KRBTEST.COM (short username oryx)
I20260809 22:37:33.950423 4020 webserver.cc:460] Webserver started at http://127.28.243.66:32867/ using document root <none> and password file <none>
I20260809 22:37:33.951135 4020 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:33.951287 4020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:33.951609 4020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:33.954779 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "d653a7f0980d485ebcb45405102b537c"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "2843edc79c3f6d4b7f2efd7d9cfb67f3"
server_key_iv: "726aec945af75c7f9b7b1a962a4a83a1"
server_key_version: "encryptionkey@0"
I20260809 22:37:33.955657 4020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "d653a7f0980d485ebcb45405102b537c"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "2843edc79c3f6d4b7f2efd7d9cfb67f3"
server_key_iv: "726aec945af75c7f9b7b1a962a4a83a1"
server_key_version: "encryptionkey@0"
I20260809 22:37:33.961390 4020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:37:33.965947 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:33.967784 4020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260809 22:37:33.967973 4020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "d653a7f0980d485ebcb45405102b537c"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "2843edc79c3f6d4b7f2efd7d9cfb67f3"
server_key_iv: "726aec945af75c7f9b7b1a962a4a83a1"
server_key_version: "encryptionkey@0"
I20260809 22:37:33.968201 4020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:33.992713 4020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:33.997612 4020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:33.997956 4020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:33.999046 4020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:34.000583 4020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:34.000664 4020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:34.000835 4020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:34.000891 4020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:34.037801 4020 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:37189
I20260809 22:37:34.037850 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:37189 every 8 connection(s)
I20260809 22:37:34.039172 4020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:34.048758 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 4020
I20260809 22:37:34.048959 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:34.049429 29645 external_mini_cluster.cc:1428] Setting key 0269c7edb61547615504d757b6d14dd9
Aug 09 22:37:34 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315053, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:34.063788 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:34.042271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:42607 (local address 127.28.243.126:40309)
0809 22:37:34.042766 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:34.042772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:34.043743 (+ 971us) server_negotiation.cc:407] Connection header received
0809 22:37:34.045694 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:34.045699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:34.045773 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:34.045889 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:34.047552 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.048388 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.050835 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.051588 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.051733 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:34.056473 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:34.056478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:34.056481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:34.056518 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:34.060002 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:34.061046 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:34.061052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:34.061054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:34.061197 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:34.061723 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:34.061727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:34.061730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:34.062056 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:34.062236 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:34.063083 (+ 847us) server_negotiation.cc:299] Negotiation successful
0809 22:37:34.063349 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260809 22:37:34.064939 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:34.042785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.66:42607)
0809 22:37:34.043573 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:34.043618 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:34.045411 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:34.046182 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:34.046190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:34.046728 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:34.047353 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:34.047375 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.048681 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.048687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:34.050674 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:34.050690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.052001 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.052010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:34.052315 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:34.053242 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:34.053277 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:34.056196 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:34.060202 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:34.060213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:34.060234 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:34.060871 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:34.061333 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:34.061339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:34.061343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:34.061573 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:34.062208 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:34.062218 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:34.062638 (+ 420us) client_negotiation.cc:769] Sending connection context
0809 22:37:34.062951 (+ 313us) client_negotiation.cc:240] Negotiation successful
0809 22:37:34.063593 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":295,"threads_started":1}
I20260809 22:37:34.066731 4157 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40309
I20260809 22:37:34.067083 4157 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:34.068002 4157 heartbeater.cc:511] Master 127.28.243.126:40309 requested a full tablet report, sending...
I20260809 22:37:34.070180 3815 ts_manager.cc:194] Registered new tserver with Master: d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189)
I20260809 22:37:34.071678 3815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:42607
WARNING: no policy specified for oryx/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab.
Entry for principal oryx/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab.
I20260809 22:37:34.117918 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40309
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab
--principal=oryx/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:34.474241 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:34.474557 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:34.474696 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:34.482323 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:34.482438 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:34.482637 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:34.493012 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab
--principal=oryx/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:40309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:34.494415 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:34.495888 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:34.510531 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:34.510727 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:34.510704 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:34.511495 4163 server_base.cc:1034] running on GCE node
I20260809 22:37:34.512142 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:34.513275 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:34.514511 4163 hybrid_clock.cc:648] HybridClock initialized: now 1786315054514423 us; error 74 us; skew 500 ppm
I20260809 22:37:34.514979 4163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:37:34.516872 4014 heartbeater.cc:503] Master 127.28.243.126:40309 was elected leader, sending a full tablet report...
Aug 09 22:37:34 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315054, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:34.522244 4163 init.cc:377] Logged in from keytab as oryx/127.28.243.67@KRBTEST.COM (short username oryx)
I20260809 22:37:34.523855 4163 webserver.cc:460] Webserver started at http://127.28.243.67:40865/ using document root <none> and password file <none>
I20260809 22:37:34.524605 4163 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:34.524735 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:34.525017 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:34.528227 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "9d5636127bbd471481f612cfc8b2cc63"
format_stamp: "Formatted at 2026-08-09 22:37:34 on dist-test-slave-rgb1"
server_key: "a01e84acbc187b92fa205cf3eeb74cee"
server_key_iv: "26018ea51ecb8e4edff28f6c851d3e3f"
server_key_version: "encryptionkey@0"
I20260809 22:37:34.529177 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "9d5636127bbd471481f612cfc8b2cc63"
format_stamp: "Formatted at 2026-08-09 22:37:34 on dist-test-slave-rgb1"
server_key: "a01e84acbc187b92fa205cf3eeb74cee"
server_key_iv: "26018ea51ecb8e4edff28f6c851d3e3f"
server_key_version: "encryptionkey@0"
I20260809 22:37:34.534793 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:37:34.539268 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:34.541275 4163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 22:37:34.541489 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "9d5636127bbd471481f612cfc8b2cc63"
format_stamp: "Formatted at 2026-08-09 22:37:34 on dist-test-slave-rgb1"
server_key: "a01e84acbc187b92fa205cf3eeb74cee"
server_key_iv: "26018ea51ecb8e4edff28f6c851d3e3f"
server_key_version: "encryptionkey@0"
I20260809 22:37:34.541735 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:34.577739 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:34.583595 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:34.583931 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:34.585101 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:34.586617 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:34.586725 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:34.586879 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:34.586963 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:34.628144 4163 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:41837
I20260809 22:37:34.628155 4300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:41837 every 8 connection(s)
I20260809 22:37:34.630121 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:34.638799 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 4163
I20260809 22:37:34.639067 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:34.639683 29645 external_mini_cluster.cc:1428] Setting key 8a34ae86963251b8d00a76d9c49d66c4
Aug 09 22:37:34 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315054, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:34.659817 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:34.635235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:45799 (local address 127.28.243.126:40309)
0809 22:37:34.635771 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:34.635778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:34.635801 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:37:34.639310 (+ 3509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:34.639317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:34.639443 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:34.639591 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:34.642040 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.643249 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.645310 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.646051 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.646183 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:34.651237 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:34.651265 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:34.651271 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:34.651323 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:34.655318 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:34.656477 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:34.656483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:34.656487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:34.656702 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:34.657306 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:34.657310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:34.657313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:34.657818 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:34.657943 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:34.659263 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0809 22:37:34.659454 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":329,"threads_started":1}
I20260809 22:37:34.660982 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:34.634556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:45799)
0809 22:37:34.635592 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:34.635650 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:34.638838 (+ 3188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:34.639883 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:34.639900 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:34.640919 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:34.641787 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:34.641822 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.643417 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.643422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:34.645138 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:34.645152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:34.646295 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:34.646303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:34.646558 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:34.647473 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:34.647513 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:34.650914 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:34.655502 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:34.655512 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:34.655534 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:34.656316 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:34.656856 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:34.656862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:34.656865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:34.657162 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:34.657953 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:34.657964 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:34.658482 (+ 518us) client_negotiation.cc:769] Sending connection context
0809 22:37:34.658906 (+ 424us) client_negotiation.cc:240] Negotiation successful
0809 22:37:34.659444 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":365,"threads_started":1}
I20260809 22:37:34.662828 4301 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40309
I20260809 22:37:34.663242 4301 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:34.664358 4301 heartbeater.cc:511] Master 127.28.243.126:40309 requested a full tablet report, sending...
I20260809 22:37:34.666402 3815 ts_manager.cc:194] Registered new tserver with Master: 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837)
I20260809 22:37:34.667701 3815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:45799
I20260809 22:37:34.670674 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.243.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.125.keytab.
Entry for principal oryx/127.28.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.125.keytab.
I20260809 22:37:34.742182 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:34161
--webserver_interface=127.28.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.125.keytab
--principal=oryx/127.28.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:40309,127.28.243.125:34161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:37:35.076267 4157 heartbeater.cc:503] Master 127.28.243.126:40309 was elected leader, sending a full tablet report...
W20260809 22:37:35.092214 4307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:35.092785 4307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:35.092998 4307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:35.101085 4307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:35.101259 4307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:35.101384 4307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:35.101495 4307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:35.101591 4307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:35.112612 4307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:40309,127.28.243.125:34161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.125.keytab
--principal=oryx/127.28.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:34161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
W20260809 22:37:35.113116 4307 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.243.126:40309,127.28.243.125:34161'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 22:37:35.114291 4307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:35.115792 4307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:35.129757 4314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:35.129768 4313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:35.130049 4316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:35.130712 4307 server_base.cc:1034] running on GCE node
I20260809 22:37:35.131628 4307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:35.133293 4307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:35.134610 4307 hybrid_clock.cc:648] HybridClock initialized: now 1786315055134518 us; error 45 us; skew 500 ppm
I20260809 22:37:35.135062 4307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:35 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:35.138787 4307 init.cc:377] Logged in from keytab as oryx/127.28.243.125@KRBTEST.COM (short username oryx)
I20260809 22:37:35.140367 4307 webserver.cc:460] Webserver started at http://127.28.243.125:37907/ using document root <none> and password file <none>
I20260809 22:37:35.141160 4307 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:35.141258 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:35.141629 4307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:35.144889 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/instance:
uuid: "93540fba9c934a9d8c4e79dc9fff7fb7"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "7894baa3080f115631f71497d70d168f"
server_key_iv: "4ba7100e33bbc35e6088124481ca303f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.145717 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal/instance:
uuid: "93540fba9c934a9d8c4e79dc9fff7fb7"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "7894baa3080f115631f71497d70d168f"
server_key_iv: "4ba7100e33bbc35e6088124481ca303f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.151448 4307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 22:37:35.155305 4323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:35.156939 4307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 22:37:35.157123 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
uuid: "93540fba9c934a9d8c4e79dc9fff7fb7"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "7894baa3080f115631f71497d70d168f"
server_key_iv: "4ba7100e33bbc35e6088124481ca303f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.157336 4307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:35.184120 4307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:35.188815 4307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:35.189145 4307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:35.209041 4307 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.125:34161
I20260809 22:37:35.209065 4379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.125:34161 every 8 connection(s)
I20260809 22:37:35.210474 4307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
I20260809 22:37:35.214632 4380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:35.219816 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 4307
I20260809 22:37:35.222455 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:37:35 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.125@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
Aug 09 22:37:35 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.125@KRBTEST.COM
I20260809 22:37:35.243903 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.225430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.28.243.126:40309)
0809 22:37:35.225959 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.225968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.226575 (+ 607us) server_negotiation.cc:407] Connection header received
0809 22:37:35.226842 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.226847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.226995 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.227104 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:35.229180 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.230234 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.232208 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.232725 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.232857 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.237341 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:35.237346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:35.237349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:35.237402 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:35.240488 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.241441 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.241449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.241453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.241601 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.242039 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.242046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.242049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.242382 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:35.242558 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.243282 (+ 724us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.243529 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":282,"threads_started":1}
I20260809 22:37:35.245082 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.222107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.28.243.125:34161)
0809 22:37:35.222918 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.222931 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.222974 (+ 43us) server_negotiation.cc:407] Connection header received
0809 22:37:35.224934 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.224961 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.225476 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.225946 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:35.228293 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.230197 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.231541 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.232423 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.232632 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.237905 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:35.237915 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:35.237928 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:35.237965 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:35.241225 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.241975 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.241984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.241987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.242133 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.242539 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.242546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.242551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.242917 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:35.243257 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.243584 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.243870 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":610,"thread_start_us":386,"threads_started":1}
I20260809 22:37:35.245126 4386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.225426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:53952)
0809 22:37:35.226402 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:35.226450 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:35.226652 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:35.227236 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:35.227245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:35.228151 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:35.228960 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.228982 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.230406 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.230409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.232046 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.232061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.232959 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.232965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.233081 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.233856 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:35.233892 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:35.237067 (+ 3175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:35.240719 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.240732 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.240745 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.241310 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.241721 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.241726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.241730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.241915 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.242519 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:35.242528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:35.242888 (+ 360us) client_negotiation.cc:769] Sending connection context
0809 22:37:35.243185 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 22:37:35.243747 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":103,"thread_start_us":508,"threads_started":1}
I20260809 22:37:35.247520 4380 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:37:35 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.125@KRBTEST.COM for oryx/127.28.243.125@KRBTEST.COM
I20260809 22:37:35.261489 4386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.248252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.0.0.1:39762)
0809 22:37:35.248508 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:35.248529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:35.248681 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:35.248973 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:35.248977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:35.249359 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:35.249492 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.249501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.250556 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.250563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.251435 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.251446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.252118 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.252124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.252348 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.253012 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:35.253030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:35.255233 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:35.258609 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.258615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.258620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.259136 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.259574 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.259582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.259589 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.259768 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.260325 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:35.260330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:35.260487 (+ 157us) client_negotiation.cc:769] Sending connection context
0809 22:37:35.260679 (+ 192us) client_negotiation.cc:240] Negotiation successful
0809 22:37:35.261027 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260809 22:37:35.261545 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.248425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.28.243.125:34161)
0809 22:37:35.248611 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.248617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.248633 (+ 16us) server_negotiation.cc:407] Connection header received
0809 22:37:35.248790 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.248795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.248861 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.248957 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:35.249659 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.250407 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.251574 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.251971 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.252110 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.255428 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:35.255436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:35.255446 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:35.255479 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:35.258301 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.259268 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.259277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.259281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.259453 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.259892 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.259900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.259904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.260201 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:35.260379 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.260772 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.261185 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":228,"server-negotiator.queue_time_us":47}
I20260809 22:37:35.267278 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Bootstrap starting.
I20260809 22:37:35.270617 4380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:35.271723 4380 log.cc:824] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:35.274907 4380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: No bootstrap required, opened a new log
I20260809 22:37:35.282141 4380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } }
I20260809 22:37:35.282490 4380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:35.282588 4380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93540fba9c934a9d8c4e79dc9fff7fb7, State: Initialized, Role: FOLLOWER
I20260809 22:37:35.283197 4380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } }
I20260809 22:37:35.284086 4388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } } }
I20260809 22:37:35.284541 4388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:37:35.285137 4380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 22:37:35.295562 4399 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:37:35.295753 4399 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:37:35.299942 4307 master_runner.cc:428] Detected that this master 93540fba9c934a9d8c4e79dc9fff7fb7 is joining an existing cluster
I20260809 22:37:35.300079 4307 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.243.125:34161
I20260809 22:37:35.331331 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.308928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.28.243.126:40309)
0809 22:37:35.309155 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.309163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.312812 (+ 3649us) server_negotiation.cc:407] Connection header received
0809 22:37:35.313164 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.313171 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.313291 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.313369 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:35.315054 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.316267 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.318711 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.319550 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.319670 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.322791 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:35.322801 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:35.322809 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:35.322863 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:35.327064 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.328171 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.328180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.328186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.328386 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.329059 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.329066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.329071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.329528 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:35.329697 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.330585 (+ 888us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.330882 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for oryx/127.28.243.124@KRBTEST.COM; defaulting to no policy
I20260809 22:37:35.332376 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.308868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:53954)
0809 22:37:35.312658 (+ 3790us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:35.312685 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:35.312985 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:35.313627 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:35.313638 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:35.314415 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:35.314848 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.314865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.316499 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.316507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.318528 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.318548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.320010 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.320018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.320251 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.321172 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:35.321201 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:35.322526 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:35.327292 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.327300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.327308 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.328002 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.328606 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.328613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.328620 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.328897 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.329799 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:35.329804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:35.330170 (+ 366us) client_negotiation.cc:769] Sending connection context
0809 22:37:35.331629 (+ 1459us) client_negotiation.cc:240] Negotiation successful
0809 22:37:35.332031 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3560,"mutex_wait_us":97,"thread_start_us":3350,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.124@KRBTEST.COM" created.
I20260809 22:37:35.344595 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.313009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.28.243.125:34161)
0809 22:37:35.313205 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.313211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.325513 (+ 12302us) server_negotiation.cc:407] Connection header received
0809 22:37:35.325946 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.325953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.326029 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.326112 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:35.327808 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.329080 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.331846 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.332788 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.332961 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.336458 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:35.336470 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:35.336474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:35.336512 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:35.341000 (+ 4488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.341836 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.341844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.341850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.342063 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.342521 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.342527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.342533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.343012 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:35.343155 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.343735 (+ 580us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.344061 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 22:37:35.346068 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.313322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.0.0.1:39764)
0809 22:37:35.325390 (+ 12068us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:35.325414 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:35.325801 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:35.326322 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:35.326327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:35.327134 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:35.327604 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.327619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.329342 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.329348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.331317 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.331333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.333024 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.333030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.333234 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.333989 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:35.334011 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:35.336290 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:35.341158 (+ 4868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.341166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.341170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.341711 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.342215 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.342222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.342228 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.342404 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.343241 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:35.343247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:35.343439 (+ 192us) client_negotiation.cc:769] Sending connection context
0809 22:37:35.345027 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0809 22:37:35.345425 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11868,"thread_start_us":12208,"threads_started":1}
Aug 09 22:37:35 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.126@KRBTEST.COM for oryx/127.28.243.125@KRBTEST.COM
I20260809 22:37:35.366016 4408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.341354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.0.0.1:39770)
0809 22:37:35.349344 (+ 7990us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:35.349392 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:35.349623 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:35.350677 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:35.350686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:35.351514 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:35.351762 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.351774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.353101 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.353108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.354248 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.354261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.355105 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.355112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.355328 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.356098 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:35.356118 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:35.358625 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:35.362691 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.362704 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.362711 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.363237 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.363701 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.363708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.363718 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.363868 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.364680 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:35.364693 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:35.364957 (+ 264us) client_negotiation.cc:769] Sending connection context
0809 22:37:35.365229 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 22:37:35.365698 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7735,"thread_start_us":7572,"threads_started":1}
I20260809 22:37:35.367041 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.341724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.28.243.125:34161)
0809 22:37:35.344982 (+ 3258us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.344991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.349478 (+ 4487us) server_negotiation.cc:407] Connection header received
0809 22:37:35.350261 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.350268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.350426 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.350617 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:35.351950 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.352950 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.354419 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.354915 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.355060 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.358843 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:35.358849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:35.358852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:35.358885 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:35.362483 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.363384 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.363391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.363395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.363571 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.364038 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.364046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.364049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.364354 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:35.364547 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.366410 (+ 1863us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.366711 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3116,"thread_start_us":8058,"threads_started":1}
I20260809 22:37:35.369263 3815 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.243.125:34161: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7"
member_type: NON_VOTER
last_known_addr {
host: "127.28.243.125"
port: 34161
}
attrs {
promote: true
}
}
dest_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333"
cas_config_opid_index: -1
I20260809 22:37:35.371105 3815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } }
I20260809 22:37:35.372327 4388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:35.372768 4388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } }
I20260809 22:37:35.374285 4388 leader_election.cc:288] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309)
I20260809 22:37:35.375453 3826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" is_pre_election: true
I20260809 22:37:35.377328 4325 leader_election.cc:302] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 93540fba9c934a9d8c4e79dc9fff7fb7; no voters: 09d0a9690b0d4e1d987e0e8cfab5b333
I20260809 22:37:35.378266 4388 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 22:37:35.379911 4414 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 93540fba9c934a9d8c4e79dc9fff7fb7 (127.28.243.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } }
I20260809 22:37:35.381106 4351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.124.keytab.
Entry for principal oryx/127.28.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.124.keytab.
I20260809 22:37:35.384253 4414 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.243.125:34161
I20260809 22:37:35.385224 4307 master.cc:537] Master@127.28.243.125:34161 shutting down...
I20260809 22:37:35.386078 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:43895
--webserver_interface=127.28.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.124.keytab
--principal=oryx/127.28.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:37:35.386669 4351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Refusing update from remote peer 09d0a9690b0d4e1d987e0e8cfab5b333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 22:37:35.398236 4413 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:35.403259 4415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } }
I20260809 22:37:35.403684 4415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:35.406867 4307 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 22:37:35.407048 3796 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260809 22:37:35.407358 4307 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:35.407603 4307 tablet_replica.cc:330] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: stopping tablet replica
W20260809 22:37:35.422039 3796 consensus_peers.cc:487] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 -> Peer 93540fba9c934a9d8c4e79dc9fff7fb7 (127.28.243.125:34161): Couldn't send request to peer 93540fba9c934a9d8c4e79dc9fff7fb7. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260809 22:37:35.430894 4307 master.cc:552] Master@127.28.243.125:34161 shutdown complete.
I20260809 22:37:35.431057 4307 master_runner.cc:305] Clearing existing system tablet
I20260809 22:37:35.431674 4307 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:35.431778 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:35.435246 4307 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260809 22:37:35.437244 4422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:35.437621 4307 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:35.437929 4307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260809 22:37:35.438024 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
uuid: "93540fba9c934a9d8c4e79dc9fff7fb7"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "7894baa3080f115631f71497d70d168f"
server_key_iv: "4ba7100e33bbc35e6088124481ca303f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.443627 4307 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:35.450891 4307 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 22:37:35.451215 4307 log.cc:1196] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 22:37:35.451683 4307 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Deleting consensus metadata
I20260809 22:37:35.451977 4307 master_runner.cc:315] Copying system tablet from 127.28.243.126:40309
I20260809 22:37:35.455034 4307 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: tablet copy: Beginning tablet copy session from remote peer at address 127.28.243.126:40309
I20260809 22:37:35.475082 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.457525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.28.243.126:40309)
0809 22:37:35.457756 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.457763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.458345 (+ 582us) server_negotiation.cc:407] Connection header received
0809 22:37:35.458772 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.458778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.458869 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.458985 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:35.460366 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.461739 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.463842 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.464711 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.464863 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.467306 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:35.467314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:35.467320 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:35.467367 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:35.471380 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.472534 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.472543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.472547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.472760 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:35.473270 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:35.473276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:35.473279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:35.473741 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:35.473916 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.474405 (+ 489us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.474674 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 22:37:35.475114 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.457079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:53966)
0809 22:37:35.458172 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:35.458209 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:35.458425 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:35.459057 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:35.459063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:35.459753 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:35.460167 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.460189 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.461956 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.461965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.463655 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.463671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.464906 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.464913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.465117 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.465904 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:35.465929 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:35.467058 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:35.471698 (+ 4640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.471705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.471710 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.472338 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.472934 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:35.472940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:35.472944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:35.473132 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:35.473904 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:35.473910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:35.474100 (+ 190us) client_negotiation.cc:769] Sending connection context
0809 22:37:35.474269 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 22:37:35.474620 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":745,"threads_started":1}
I20260809 22:37:35.476538 3836 tablet_copy_service.cc:140] P 09d0a9690b0d4e1d987e0e8cfab5b333: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 93540fba9c934a9d8c4e79dc9fff7fb7 ({username='oryx', principal='oryx/127.28.243.125@KRBTEST.COM'} at 127.0.0.1:53966)
I20260809 22:37:35.476994 3836 tablet_copy_service.cc:161] P 09d0a9690b0d4e1d987e0e8cfab5b333: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 93540fba9c934a9d8c4e79dc9fff7fb7 at {username='oryx', principal='oryx/127.28.243.125@KRBTEST.COM'} at 127.0.0.1:53966: session id = 93540fba9c934a9d8c4e79dc9fff7fb7-00000000000000000000000000000000
I20260809 22:37:35.482060 3836 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 22:37:35.485316 4307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 22:37:35.494287 4307 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: tablet copy: Starting download of 0 data blocks...
I20260809 22:37:35.494887 4307 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: tablet copy: Starting download of 1 WAL segments...
I20260809 22:37:35.499369 4307 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 22:37:35.505470 3836 tablet_copy_service.cc:342] P 09d0a9690b0d4e1d987e0e8cfab5b333: Request end of tablet copy session 93540fba9c934a9d8c4e79dc9fff7fb7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.243.125@KRBTEST.COM'} at 127.0.0.1:53966
I20260809 22:37:35.505743 3836 tablet_copy_service.cc:433] P 09d0a9690b0d4e1d987e0e8cfab5b333: ending tablet copy session 93540fba9c934a9d8c4e79dc9fff7fb7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 93540fba9c934a9d8c4e79dc9fff7fb7
I20260809 22:37:35.518587 4307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:35.522279 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:35.530715 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:35.538887 4434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:35.539330 4307 server_base.cc:1034] running on GCE node
I20260809 22:37:35.539865 4307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:35.540838 4307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:35.542181 4307 hybrid_clock.cc:648] HybridClock initialized: now 1786315055542100 us; error 49 us; skew 500 ppm
I20260809 22:37:35.542485 4307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:35 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:35.545069 4307 init.cc:377] Logged in from keytab as oryx/127.28.243.125@KRBTEST.COM (short username oryx)
I20260809 22:37:35.552021 4307 webserver.cc:460] Webserver started at http://127.28.243.125:46631/ using document root <none> and password file <none>
I20260809 22:37:35.552490 4307 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:35.552611 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:35.556818 4307 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260809 22:37:35.560031 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:35.560694 4307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260809 22:37:35.560865 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
uuid: "93540fba9c934a9d8c4e79dc9fff7fb7"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "7894baa3080f115631f71497d70d168f"
server_key_iv: "4ba7100e33bbc35e6088124481ca303f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.561069 4307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:35.578619 4307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:35.583467 4307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:35.613265 4307 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.125:34161
I20260809 22:37:35.613320 4494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.125:34161 every 8 connection(s)
I20260809 22:37:35.614434 4307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
I20260809 22:37:35.625046 4495 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:37:35.629292 4495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Bootstrap starting.
I20260809 22:37:35.647521 4495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:37:35.648166 4495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: Bootstrap complete.
I20260809 22:37:35.649260 4495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } }
I20260809 22:37:35.649536 4495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93540fba9c934a9d8c4e79dc9fff7fb7, State: Initialized, Role: LEARNER
I20260809 22:37:35.649792 4495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } }
I20260809 22:37:35.650727 4498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } }
I20260809 22:37:35.651152 4498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: LEARNER
I20260809 22:37:35.651150 4495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:35.663519 4510 catalog_manager.cc:1259] Loaded cluster ID: 92f5287dfbb847b7942995cc5466fb95
I20260809 22:37:35.663758 4510 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: loading cluster ID for follower catalog manager: success
I20260809 22:37:35.666425 4510 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: acquiring CA information for follower catalog manager: success
I20260809 22:37:35.668485 4510 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:37:35.671419 4301 heartbeater.cc:503] Master 127.28.243.126:40309 was elected leader, sending a full tablet report...
W20260809 22:37:35.850399 4416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:35.850806 4416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:35.850994 4416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:35.860059 4416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:35.860335 4416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:35.860443 4416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:35.860585 4416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:35.860781 4416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:35.872313 4416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.124.keytab
--principal=oryx/127.28.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:43895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:35.874010 4416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:35.875588 4416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:35.895008 4516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:35.897395 4517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:35.900895 4416 server_base.cc:1034] running on GCE node
W20260809 22:37:35.902348 4521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:35.903203 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.884829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.0.0.1:39776)
0809 22:37:35.895766 (+ 10937us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:35.895789 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:35.895962 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:35.896784 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:35.896792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:35.896828 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:35.897282 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.897303 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.899386 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.899392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.900788 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:35.900801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.902374 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.902380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:35.902620 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.902693 (+ 73us) client_negotiation.cc:769] Sending connection context
0809 22:37:35.902825 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 22:37:35.902969 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10777,"thread_start_us":10733,"threads_started":1}
I20260809 22:37:35.903378 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.885107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.28.243.125:34161)
0809 22:37:35.896269 (+ 11162us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.896278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.896306 (+ 28us) server_negotiation.cc:407] Connection header received
0809 22:37:35.896408 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.896417 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.896540 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.896694 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:35.897617 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.899221 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.900949 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:35.902147 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:35.902373 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:35.902585 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:35.902863 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 22:37:35.903019 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11003,"thread_start_us":10862,"threads_started":1}
I20260809 22:37:35.903741 4416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:35.904636 4468 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 22:37:35.905270 4498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d0a9690b0d4e1d987e0e8cfab5b333. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } }
I20260809 22:37:35.905555 4498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: LEARNER
I20260809 22:37:35.906436 4416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:35.907744 4416 hybrid_clock.cc:648] HybridClock initialized: now 1786315055907666 us; error 42 us; skew 500 ppm
I20260809 22:37:35.908236 4416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:35 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:35.911891 4416 init.cc:377] Logged in from keytab as oryx/127.28.243.124@KRBTEST.COM (short username oryx)
I20260809 22:37:35.914162 4416 webserver.cc:460] Webserver started at http://127.28.243.124:45185/ using document root <none> and password file <none>
I20260809 22:37:35.914964 4416 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:35.915118 4416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:35.915398 4416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:35.919121 4416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/instance:
uuid: "e87b4c7179b740b68a5758353cd69f96"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "6026148fe9bdb05ab4808ddfc2a79f3d"
server_key_iv: "3294fbf9fc9c67ee89d819ae94941b9f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.919968 4416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal/instance:
uuid: "e87b4c7179b740b68a5758353cd69f96"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "6026148fe9bdb05ab4808ddfc2a79f3d"
server_key_iv: "3294fbf9fc9c67ee89d819ae94941b9f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.925773 4416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260809 22:37:35.929764 4528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:35.931396 4416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:37:35.931612 4416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
uuid: "e87b4c7179b740b68a5758353cd69f96"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "6026148fe9bdb05ab4808ddfc2a79f3d"
server_key_iv: "3294fbf9fc9c67ee89d819ae94941b9f"
server_key_version: "encryptionkey@0"
I20260809 22:37:35.931793 4416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:35.951560 4416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:35.956360 4416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:35.956712 4416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:35.981957 4416 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.124:43895
I20260809 22:37:35.982033 4580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.124:43895 every 8 connection(s)
I20260809 22:37:35.983562 4416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
I20260809 22:37:35.988322 4581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:35.990469 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 4416
I20260809 22:37:36.000063 4581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.124@KRBTEST.COM
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.124@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:36.020743 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:35.993480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.28.243.124:43895)
0809 22:37:35.994678 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:35.994691 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:35.994738 (+ 47us) server_negotiation.cc:407] Connection header received
0809 22:37:35.996921 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:35.996947 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:35.997417 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:35.997912 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.000267 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.002061 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.003795 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.004970 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.005207 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.010365 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.010378 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.010393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.010443 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.015109 (+ 4666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.016071 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.016081 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.016091 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.016292 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.016933 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.016939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.016942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.017835 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.018169 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.018681 (+ 512us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.019186 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":617,"threads_started":1}
I20260809 22:37:36.022401 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.003261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.28.243.126:40309)
0809 22:37:36.004786 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.004790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.004807 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:37:36.004858 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.004862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.004924 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.005185 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.006693 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.007413 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.009267 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.010011 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.010173 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.014621 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.014627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.014634 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.014685 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.018516 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.019556 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.019563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.019566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.019851 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.020449 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.020462 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.020465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.021088 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.021255 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.021799 (+ 544us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.021985 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1420,"thread_start_us":1302,"threads_started":1}
I20260809 22:37:36.022423 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.002979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:53970)
0809 22:37:36.003887 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.003941 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.004307 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.005259 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.005267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.006233 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.006525 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.006534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.007574 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.007586 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.009106 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.009121 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.010378 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.010386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.010641 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.011434 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.011514 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.014379 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.018696 (+ 4317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.018705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.018713 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.019315 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.020054 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.020059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.020062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.020236 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.021223 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.021231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.021436 (+ 205us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.021672 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.022013 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":366,"threads_started":1}
I20260809 22:37:36.024127 4581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.124@KRBTEST.COM for oryx/127.28.243.125@KRBTEST.COM
I20260809 22:37:36.038941 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.024863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.0.0.1:39790)
0809 22:37:36.025040 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.025061 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.025189 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.025551 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.025556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.025914 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.026053 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.026060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.027065 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.027071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.028020 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.028033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.028857 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.028863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.029084 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.029892 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.029909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.032147 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.036152 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.036158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.036162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.036720 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.037153 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.037161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.037165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.037361 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.037928 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.037933 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.038083 (+ 150us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.038239 (+ 156us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.038596 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 22:37:36.038947 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.025070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.28.243.125:34161)
0809 22:37:36.025270 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.025278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.025295 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:37:36.025353 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.025357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.025419 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.025523 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.026196 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.026922 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.028174 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.028697 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.028834 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.032395 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.032401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.032405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.032440 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.035964 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.036849 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.036856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.036860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.037014 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.037492 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.037499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.037506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.037792 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.037952 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.038344 (+ 392us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.038594 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 22:37:36.040167 4581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315055, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.124@KRBTEST.COM for oryx/127.28.243.124@KRBTEST.COM
I20260809 22:37:36.054958 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.040796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.0.0.1:57204)
0809 22:37:36.041148 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.041171 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.041291 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.041629 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.041634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.042100 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.042298 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.042312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.043355 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.043364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.044268 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.044279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.045098 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.045104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.045374 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.046089 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.046112 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.048821 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.052440 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.052444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.052455 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.052959 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.053392 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.053395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.053398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.053570 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.054162 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.054165 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.054314 (+ 149us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.054433 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.054629 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":22}
I20260809 22:37:36.055182 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.041056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.28.243.124:43895)
0809 22:37:36.041223 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.041229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.041247 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:36.041430 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.041442 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.041499 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.041612 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.042457 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.043217 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.044421 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.044933 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.045086 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.049055 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.049060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.049064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.049096 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.052279 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.053090 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.053095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.053097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.053243 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.053704 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.053708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.053711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.054040 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.054177 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.054568 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.054779 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":55}
I20260809 22:37:36.060844 4581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Bootstrap starting.
I20260809 22:37:36.064529 4581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:36.065728 4581 log.cc:824] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:36.069063 4581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: No bootstrap required, opened a new log
I20260809 22:37:36.077345 4581 raft_consensus.cc:348] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } }
I20260809 22:37:36.077734 4581 raft_consensus.cc:374] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:36.078208 4581 raft_consensus.cc:729] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87b4c7179b740b68a5758353cd69f96, State: Initialized, Role: FOLLOWER
I20260809 22:37:36.079006 4581 consensus_queue.cc:260] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } }
I20260809 22:37:36.080163 4589 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } } }
I20260809 22:37:36.080610 4589 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:37:36.081423 4581 sys_catalog.cc:564] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:36.087539 4601 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:37:36.097868 4416 master_runner.cc:428] Detected that this master e87b4c7179b740b68a5758353cd69f96 is joining an existing cluster
I20260809 22:37:36.098102 4416 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.243.124:43895
I20260809 22:37:36.120396 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.102497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.28.243.126:40309)
0809 22:37:36.102695 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.102700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.103352 (+ 652us) server_negotiation.cc:407] Connection header received
0809 22:37:36.103865 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.103871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.104051 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.104174 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.105507 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.106730 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.108649 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.109322 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.109482 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.112740 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.112748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.112753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.112795 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.116223 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.117222 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.117231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.117238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.117411 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.118039 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.118047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.118051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.118498 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.119016 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.119675 (+ 659us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.119972 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:37:36.120661 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.102284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:53986)
0809 22:37:36.103064 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.103097 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.103374 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.104396 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.104401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.105048 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.105350 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.105358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.106872 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.106877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.108398 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.108423 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.109535 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.109544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.109816 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.110976 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.111031 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.112473 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.116377 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.116389 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.116393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.116964 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.117567 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.117572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.117580 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.117880 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.118968 (+ 1088us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.118974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.119281 (+ 307us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.119508 (+ 227us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.119916 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":407,"threads_started":1}
I20260809 22:37:36.121037 4520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.103699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.28.243.125:34161)
0809 22:37:36.103871 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.103876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.104311 (+ 435us) server_negotiation.cc:407] Connection header received
0809 22:37:36.104651 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.104657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.104745 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.104819 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.106568 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.107355 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.109526 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.110137 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.110475 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.112803 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.112820 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.112826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.112875 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.117552 (+ 4677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.118421 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.118427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.118432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.118601 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.119196 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.119201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.119205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.119555 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.119693 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.120352 (+ 659us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.120669 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 22:37:36.121858 4607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.103384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.0.0.1:39798)
0809 22:37:36.104022 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.104044 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.104444 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.105069 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.105076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.106035 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.106418 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.106429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.107500 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.107507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.109360 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.109377 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.110499 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.110507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.110723 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.111381 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.111405 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.112499 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.117696 (+ 5197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.117705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.117708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.118277 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.118754 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.118759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.118763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.119064 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.119778 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.119782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.120021 (+ 239us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.121240 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.121547 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":26,"spinlock_wait_cycles":22912,"thread_start_us":330,"threads_started":1}
I20260809 22:37:36.123696 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.104045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.0.0.1:57210)
0809 22:37:36.106061 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.106089 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.106235 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.107511 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.107514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.108040 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.108345 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.108353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.109715 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.109720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.111461 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.111483 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.112995 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.113004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.113191 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.113832 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.113859 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.115081 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.119164 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.119172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.119177 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.119718 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.120252 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.120257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.120262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.120446 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.121870 (+ 1424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.121874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.122084 (+ 210us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.122243 (+ 159us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.122882 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1878,"thread_start_us":1677,"threads_started":1}
I20260809 22:37:36.123961 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.104388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57210 (local address 127.28.243.124:43895)
0809 22:37:36.104632 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.104638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.106207 (+ 1569us) server_negotiation.cc:407] Connection header received
0809 22:37:36.106349 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.106356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.106443 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.106531 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.108498 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.109493 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.111659 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.112511 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.112710 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.115266 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.115279 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.115283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.115311 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.119030 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.119884 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.119889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.119892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.120116 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.120677 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.120686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.120692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.121192 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.121771 (+ 579us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.122458 (+ 687us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.123211 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":341,"server-negotiator.queue_time_us":77,"spinlock_wait_cycles":52480}
I20260809 22:37:36.128063 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 3877
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.126@KRBTEST.COM for oryx/127.28.243.124@KRBTEST.COM
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.65@KRBTEST.COM
I20260809 22:37:36.144680 4519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.128410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.0.0.1:57216)
0809 22:37:36.128659 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.128682 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.128798 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.129226 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.129232 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.129803 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.129963 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.129971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.131165 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.131170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.132411 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.132459 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.133465 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.133475 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.133694 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.134362 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.134381 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.137448 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.141441 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.141450 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.141458 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.142142 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.142733 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.142738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.142740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.142920 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.143741 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.143752 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.143919 (+ 167us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.144103 (+ 184us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.144341 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 22:37:36.144891 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.128493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.28.243.124:43895)
0809 22:37:36.128698 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.128703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.128723 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:36.128973 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.128978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.129076 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.129194 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.130130 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.131004 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.132699 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.133264 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.133373 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.137700 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.137705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.137708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.137742 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.141266 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.142375 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.142381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.142385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.142575 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.143058 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.143065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.143070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.143575 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.143778 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.144271 (+ 493us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.144517 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 22:37:36.146601 3815 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.243.124:43895: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e87b4c7179b740b68a5758353cd69f96"
member_type: NON_VOTER
last_known_addr {
host: "127.28.243.124"
port: 43895
}
attrs {
promote: true
}
}
dest_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333"
cas_config_opid_index: 5
I20260809 22:37:36.147686 3815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } }
I20260809 22:37:36.149642 4415 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e87b4c7179b740b68a5758353cd69f96 (127.28.243.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.150317 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.129306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.28.243.65:32887)
0809 22:37:36.130087 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.130098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.130172 (+ 74us) server_negotiation.cc:407] Connection header received
0809 22:37:36.130240 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.130247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.130458 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.130596 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.132118 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.133416 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.135205 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.136419 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.136608 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.142523 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.142544 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.142565 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.142613 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.146731 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.147550 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.147565 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.147567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.147711 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.148212 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.148217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.148220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.148706 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.149082 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.149410 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.149714 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":436,"threads_started":1}
I20260809 22:37:36.150447 4468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 LEARNER]: Refusing update from remote peer 09d0a9690b0d4e1d987e0e8cfab5b333: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 22:37:36.150889 4556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:36.151880 4008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:36.153820 4415 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.243.124:43895
I20260809 22:37:36.154614 4416 master.cc:537] Master@127.28.243.124:43895 shutting down...
I20260809 22:37:36.158711 4612 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 22:37:36.161916 4612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.162390 4612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:36.163065 4468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e87b4c7179b740b68a5758353cd69f96 (127.28.243.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.166710 4498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: NON_VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: true } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.167117 4498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: LEARNER
I20260809 22:37:36.167737 4612 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: attempting to promote NON_VOTER 93540fba9c934a9d8c4e79dc9fff7fb7 to VOTER
I20260809 22:37:36.168478 4612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } }
I20260809 22:37:36.170619 4468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 LEARNER]: Refusing update from remote peer 09d0a9690b0d4e1d987e0e8cfab5b333: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 22:37:36.171546 4415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 22:37:36.175273 4611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 93540fba9c934a9d8c4e79dc9fff7fb7 (127.28.243.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.175944 4468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 93540fba9c934a9d8c4e79dc9fff7fb7 (127.28.243.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.178519 4556 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Refusing update from remote peer 09d0a9690b0d4e1d987e0e8cfab5b333: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 22:37:36.180187 4611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.180826 4498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.181094 4498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:37:36.181306 4611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:36.181633 4611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.182790 4611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:36.191044 4416 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:36.191478 4416 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:36.191694 4416 tablet_replica.cc:330] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: stopping tablet replica
W20260809 22:37:36.192706 3796 consensus_peers.cc:487] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 -> Peer e87b4c7179b740b68a5758353cd69f96 (127.28.243.124:43895): Couldn't send request to peer e87b4c7179b740b68a5758353cd69f96. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 22:37:36.208436 4416 master.cc:552] Master@127.28.243.124:43895 shutdown complete.
I20260809 22:37:36.208550 4416 master_runner.cc:305] Clearing existing system tablet
I20260809 22:37:36.209061 4416 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:36.209177 4416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:36.212042 4416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 22:37:36.213716 4618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:36.214176 4416 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:36.214459 4416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260809 22:37:36.214576 4416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
uuid: "e87b4c7179b740b68a5758353cd69f96"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "6026148fe9bdb05ab4808ddfc2a79f3d"
server_key_iv: "3294fbf9fc9c67ee89d819ae94941b9f"
server_key_version: "encryptionkey@0"
I20260809 22:37:36.218372 4416 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:36.223402 4416 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 22:37:36.223580 4416 log.cc:1196] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 22:37:36.224038 4416 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Deleting consensus metadata
I20260809 22:37:36.224342 4416 master_runner.cc:315] Copying system tablet from 127.28.243.126:40309
I20260809 22:37:36.227218 4416 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: tablet copy: Beginning tablet copy session from remote peer at address 127.28.243.126:40309
I20260809 22:37:36.242121 4625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.227995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54000)
0809 22:37:36.228665 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.228692 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.228871 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.229475 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.229479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.229972 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.230188 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.230197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.231765 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.231770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.232871 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.232885 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.233890 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.233897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.234109 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.234839 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.234872 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.235802 (+ 930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.239733 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.239737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.239740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.240126 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.240516 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.240519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.240522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.240725 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.241315 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.241319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.241470 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.241598 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.241825 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":554,"threads_started":1}
I20260809 22:37:36.242322 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.228236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.28.243.126:40309)
0809 22:37:36.228508 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.228515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.228805 (+ 290us) server_negotiation.cc:407] Connection header received
0809 22:37:36.228998 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.229004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.229108 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.229215 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.230401 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.231594 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.233030 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.233696 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.233834 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.235993 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.236000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.236005 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.236045 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.239570 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.240251 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.240257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.240261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.240407 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.240833 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.240838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.240841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.241193 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.241346 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.241749 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.241979 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260809 22:37:36.242938 3836 tablet_copy_service.cc:140] P 09d0a9690b0d4e1d987e0e8cfab5b333: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e87b4c7179b740b68a5758353cd69f96 ({username='oryx', principal='oryx/127.28.243.124@KRBTEST.COM'} at 127.0.0.1:54000)
I20260809 22:37:36.243186 3836 tablet_copy_service.cc:161] P 09d0a9690b0d4e1d987e0e8cfab5b333: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e87b4c7179b740b68a5758353cd69f96 at {username='oryx', principal='oryx/127.28.243.124@KRBTEST.COM'} at 127.0.0.1:54000: session id = e87b4c7179b740b68a5758353cd69f96-00000000000000000000000000000000
I20260809 22:37:36.245640 3836 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 22:37:36.247460 4416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 22:37:36.251952 4416 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: tablet copy: Starting download of 0 data blocks...
I20260809 22:37:36.252244 4416 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: tablet copy: Starting download of 1 WAL segments...
I20260809 22:37:36.254902 4416 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 22:37:36.258695 3836 tablet_copy_service.cc:342] P 09d0a9690b0d4e1d987e0e8cfab5b333: Request end of tablet copy session e87b4c7179b740b68a5758353cd69f96-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.243.124@KRBTEST.COM'} at 127.0.0.1:54000
I20260809 22:37:36.258833 3836 tablet_copy_service.cc:433] P 09d0a9690b0d4e1d987e0e8cfab5b333: ending tablet copy session e87b4c7179b740b68a5758353cd69f96-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e87b4c7179b740b68a5758353cd69f96
I20260809 22:37:36.269002 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 3877
I20260809 22:37:36.271816 4416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:36.275851 4626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:36.276988 4627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:36.278702 4416 server_base.cc:1034] running on GCE node
W20260809 22:37:36.278968 4630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:36.279371 4416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:36.280125 4416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:36.281361 4416 hybrid_clock.cc:648] HybridClock initialized: now 1786315056281296 us; error 60 us; skew 500 ppm
I20260809 22:37:36.281626 4416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315056, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:36.283967 4416 init.cc:377] Logged in from keytab as oryx/127.28.243.124@KRBTEST.COM (short username oryx)
I20260809 22:37:36.285344 4416 webserver.cc:460] Webserver started at http://127.28.243.124:35991/ using document root <none> and password file <none>
I20260809 22:37:36.285707 4416 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:36.285818 4416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:36.289376 4416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260809 22:37:36.291368 4636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:36.291846 4416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260809 22:37:36.291983 4416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
uuid: "e87b4c7179b740b68a5758353cd69f96"
format_stamp: "Formatted at 2026-08-09 22:37:35 on dist-test-slave-rgb1"
server_key: "6026148fe9bdb05ab4808ddfc2a79f3d"
server_key_iv: "3294fbf9fc9c67ee89d819ae94941b9f"
server_key_version: "encryptionkey@0"
I20260809 22:37:36.292182 4416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:36.297307 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:32887
--local_ip_for_outbound_sockets=127.28.243.65
--tserver_master_addrs=127.28.243.126:40309
--webserver_port=38029
--webserver_interface=127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab
--principal=oryx/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:37:36.310642 4416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:36.314888 4416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:36.333573 4416 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.124:43895
I20260809 22:37:36.333592 4694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.124:43895 every 8 connection(s)
I20260809 22:37:36.334592 4416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
I20260809 22:37:36.338536 4695 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:37:36.341053 4695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Bootstrap starting.
I20260809 22:37:36.354806 4695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:37:36.355237 4695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Bootstrap complete.
I20260809 22:37:36.356076 4695 raft_consensus.cc:348] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } }
I20260809 22:37:36.356324 4695 raft_consensus.cc:729] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e87b4c7179b740b68a5758353cd69f96, State: Initialized, Role: LEARNER
I20260809 22:37:36.356616 4695 consensus_queue.cc:260] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } }
I20260809 22:37:36.357465 4698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.357784 4695 sys_catalog.cc:564] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:36.357788 4698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: This master's current role is: LEARNER
I20260809 22:37:36.369658 4709 catalog_manager.cc:1259] Loaded cluster ID: 92f5287dfbb847b7942995cc5466fb95
I20260809 22:37:36.369789 4709 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: loading cluster ID for follower catalog manager: success
I20260809 22:37:36.373517 4709 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: acquiring CA information for follower catalog manager: success
I20260809 22:37:36.375975 4709 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 22:37:36.686060 4641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:36.686538 4641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:36.686707 4641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 22:37:36.692947 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.676077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.28.243.124:43895)
0809 22:37:36.684035 (+ 7958us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.684042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.684309 (+ 267us) server_negotiation.cc:407] Connection header received
0809 22:37:36.684533 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.684538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.684661 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.684801 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:36.685873 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.687707 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.690150 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.691667 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.691907 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.692125 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.692355 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.692510 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7766,"thread_start_us":7604,"threads_started":1}
I20260809 22:37:36.692955 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.675807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.0.0.1:57224)
0809 22:37:36.684150 (+ 8343us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.684173 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.684369 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.684864 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.684872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.684904 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:36.685619 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.685634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.687873 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.687879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.689949 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.689967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.691954 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.691962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.692159 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.692198 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.692315 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.692481 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8223,"thread_start_us":8122,"threads_started":1}
I20260809 22:37:36.694636 4668 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 22:37:36.695422 4698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09d0a9690b0d4e1d987e0e8cfab5b333. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:36.695725 4698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: This master's current role is: LEARNER
W20260809 22:37:36.697438 4641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:36.697588 4641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:36.697765 4641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:36.708182 4641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab
--principal=oryx/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:32887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=38029
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:36.709718 4641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:36.711092 4641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:36.726601 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:36.726748 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:36.727283 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:36.728339 4641 server_base.cc:1034] running on GCE node
I20260809 22:37:36.729169 4641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:36.730252 4641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:36.731500 4641 hybrid_clock.cc:648] HybridClock initialized: now 1786315056731427 us; error 63 us; skew 500 ppm
I20260809 22:37:36.731838 4641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315056, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:36.735594 4641 init.cc:377] Logged in from keytab as oryx/127.28.243.65@KRBTEST.COM (short username oryx)
I20260809 22:37:36.737295 4641 webserver.cc:460] Webserver started at http://127.28.243.65:38029/ using document root <none> and password file <none>
I20260809 22:37:36.738070 4641 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:36.738224 4641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:36.744311 4641 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 22:37:36.747376 4727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:36.749109 4641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:36.749300 4641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "6ef923bb58564adc9eadd9c9e9cf1346"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "c425e0cedf18d04333ab72fd92401c54"
server_key_iv: "937f52b7d41b980842300a0fbb089f63"
server_key_version: "encryptionkey@0"
I20260809 22:37:36.750399 4641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:36.790568 4641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:36.796226 4641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:36.796602 4641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:36.797857 4641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:36.799657 4641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:36.799784 4641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:36.799906 4641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:36.799976 4641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:36.840386 4641 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:32887
I20260809 22:37:36.840420 4845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:32887 every 8 connection(s)
I20260809 22:37:36.842212 4641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:36.852933 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 4641
I20260809 22:37:36.853363 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 4020
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315056, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.125@KRBTEST.COM
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315056, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.124@KRBTEST.COM
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315056, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:36.874604 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.848650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:43873 (local address 127.28.243.126:40309)
0809 22:37:36.849616 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.849622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.849643 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:37:36.850484 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.850489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.850563 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.850681 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.853678 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.854872 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.858395 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.859160 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.859270 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.866771 (+ 7501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.866779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.866784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.866836 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.871001 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.871950 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.871956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.871960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.872135 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.872689 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.872694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.872697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.873062 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.873211 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.873920 (+ 709us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.874176 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":725,"threads_started":1}
I20260809 22:37:36.875969 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.846117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.28.243.65:57079)
0809 22:37:36.847006 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.847053 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.850057 (+ 3004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.851032 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.851038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.852077 (+ 1039us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.853458 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.853485 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.855201 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.855209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.858259 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.858279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.860037 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.860046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.860376 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.861494 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.861535 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.865896 (+ 4361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.870421 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.870430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.870448 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.870919 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.871435 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.871441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.871445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.871783 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.872794 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.872805 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.873310 (+ 505us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.873746 (+ 436us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.874323 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":366,"spinlock_wait_cycles":18816,"thread_start_us":373,"threads_started":1}
I20260809 22:37:36.876003 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.848460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.65:43873)
0809 22:37:36.849134 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.849154 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.850063 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.851016 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.851026 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.852077 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.853456 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.853484 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.855181 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.855185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.858201 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.858217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.859645 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.859653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.860139 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.861275 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.861314 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.866394 (+ 5080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.871119 (+ 4725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.871123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.871126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.871796 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.872339 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.872343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.872347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.872527 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.873293 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.873299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.873553 (+ 254us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.874895 (+ 1342us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.875228 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":602,"spinlock_wait_cycles":8064,"thread_start_us":387,"threads_started":1}
I20260809 22:37:36.877040 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.847610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.28.243.65:48343)
0809 22:37:36.848431 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:36.848449 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:36.850050 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:36.851609 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:36.851613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:36.852206 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:36.853462 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.853484 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.855090 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.855095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.858201 (+ 3106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:36.858217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.859645 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.859653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:36.860145 (+ 492us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.861329 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:36.861346 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:36.866522 (+ 5176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:36.874057 (+ 7535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.874063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.874067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.874564 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.875038 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:36.875042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:36.875045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:36.875229 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:36.875973 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:36.875978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:36.876124 (+ 146us) client_negotiation.cc:769] Sending connection context
0809 22:37:36.876291 (+ 167us) client_negotiation.cc:240] Negotiation successful
0809 22:37:36.876594 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":564,"spinlock_wait_cycles":1280,"thread_start_us":423,"threads_started":1}
I20260809 22:37:36.877082 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.847836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:48343 (local address 127.28.243.125:34161)
0809 22:37:36.851224 (+ 3388us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.851231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.851259 (+ 28us) server_negotiation.cc:407] Connection header received
0809 22:37:36.851321 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.851329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.851421 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.851560 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.853678 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.854903 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.858404 (+ 3501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.859160 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.859281 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.866789 (+ 7508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.866796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.866801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.866844 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.873814 (+ 6970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.874722 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.874729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.874732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.874878 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.875365 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.875369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.875372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.875732 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.875886 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.876439 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.876713 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3258,"thread_start_us":3109,"threads_started":1}
I20260809 22:37:36.877235 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.845608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:57079 (local address 127.28.243.124:43895)
0809 22:37:36.845821 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.845828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.847186 (+ 1358us) server_negotiation.cc:407] Connection header received
0809 22:37:36.850411 (+ 3225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.850419 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.850512 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.850603 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.853692 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.854860 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.858542 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.859487 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.859620 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.866247 (+ 6627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.866256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.866261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.866311 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.870255 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.871057 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.871063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.871067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.871305 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.871925 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.871934 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.871939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.872402 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.872698 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.876543 (+ 3845us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.876827 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 22:37:36.878369 4847 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40309
I20260809 22:37:36.878525 4846 heartbeater.cc:347] Connected to a master server at 127.28.243.124:43895
I20260809 22:37:36.878782 4847 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:36.878788 4846 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:36.878417 4848 heartbeater.cc:347] Connected to a master server at 127.28.243.125:34161
I20260809 22:37:36.879426 4848 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:36.879969 4846 heartbeater.cc:511] Master 127.28.243.124:43895 requested a full tablet report, sending...
I20260809 22:37:36.880024 4848 heartbeater.cc:511] Master 127.28.243.125:34161 requested a full tablet report, sending...
I20260809 22:37:36.880208 4847 heartbeater.cc:511] Master 127.28.243.126:40309 requested a full tablet report, sending...
I20260809 22:37:36.882175 3815 ts_manager.cc:194] Re-registered known tserver with Master: 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887)
I20260809 22:37:36.882753 4458 ts_manager.cc:194] Registered new tserver with Master: 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887)
I20260809 22:37:36.882833 4655 ts_manager.cc:194] Registered new tserver with Master: 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887)
I20260809 22:37:36.883329 3815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:43873
Aug 09 22:37:36 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.66@KRBTEST.COM
I20260809 22:37:36.891726 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:36.854740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.28.243.66:37189)
0809 22:37:36.865904 (+ 11164us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:36.865918 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:36.866755 (+ 837us) server_negotiation.cc:407] Connection header received
0809 22:37:36.866829 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:36.866838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:36.867165 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:36.867329 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:36.870074 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.871646 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.873784 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:36.874660 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:36.875079 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:36.885142 (+ 10063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:36.885156 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:36.885168 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:36.885205 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:36.888877 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.889680 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.889687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.889690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.889893 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:36.890315 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:36.890319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:36.890321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:36.890697 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:36.890976 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:36.891213 (+ 237us) server_negotiation.cc:299] Negotiation successful
0809 22:37:36.891421 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10912,"thread_start_us":10034,"threads_started":1}
I20260809 22:37:36.893249 4152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:36.982113 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 4020
I20260809 22:37:37.002156 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:37189
--local_ip_for_outbound_sockets=127.28.243.66
--tserver_master_addrs=127.28.243.126:40309
--webserver_port=32867
--webserver_interface=127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab
--principal=oryx/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:37:37.261639 4859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: NON_VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: true } } }
I20260809 22:37:37.262441 4859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:37.263537 4861 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: attempting to promote NON_VOTER e87b4c7179b740b68a5758353cd69f96 to VOTER
I20260809 22:37:37.264616 4861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } }
I20260809 22:37:37.267799 4668 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 LEARNER]: Refusing update from remote peer 09d0a9690b0d4e1d987e0e8cfab5b333: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 22:37:37.268821 4468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Refusing update from remote peer 09d0a9690b0d4e1d987e0e8cfab5b333: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 22:37:37.269649 4864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 22:37:37.270176 4861 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 22:37:37.273188 4861 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e87b4c7179b740b68a5758353cd69f96 (127.28.243.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.273962 4468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e87b4c7179b740b68a5758353cd69f96 (127.28.243.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.277171 4859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.277443 4859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:37.277848 4859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.278185 4859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:37.281076 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.281525 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:37:37.283425 4861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.283715 4861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:37.284137 4668 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e87b4c7179b740b68a5758353cd69f96 (127.28.243.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.287726 4868 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:37.288082 4868 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: This master's current role is: FOLLOWER
W20260809 22:37:37.364917 4858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:37.365227 4858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:37.365324 4858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:37.373030 4858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:37.373164 4858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:37.373281 4858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:37.384078 4858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab
--principal=oryx/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:37189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=32867
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:37.385629 4858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:37.387035 4858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:37.405741 4876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:37.405910 4874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:37.406509 4873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:37.407897 4858 server_base.cc:1034] running on GCE node
I20260809 22:37:37.408645 4858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:37.409703 4858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:37.411041 4858 hybrid_clock.cc:648] HybridClock initialized: now 1786315057410980 us; error 54 us; skew 500 ppm
I20260809 22:37:37.411386 4858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:37 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:37.415119 4858 init.cc:377] Logged in from keytab as oryx/127.28.243.66@KRBTEST.COM (short username oryx)
I20260809 22:37:37.416815 4858 webserver.cc:460] Webserver started at http://127.28.243.66:32867/ using document root <none> and password file <none>
I20260809 22:37:37.417573 4858 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:37.417768 4858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:37.423836 4858 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 22:37:37.426995 4883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:37.428846 4858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 22:37:37.429024 4858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "d653a7f0980d485ebcb45405102b537c"
format_stamp: "Formatted at 2026-08-09 22:37:33 on dist-test-slave-rgb1"
server_key: "2843edc79c3f6d4b7f2efd7d9cfb67f3"
server_key_iv: "726aec945af75c7f9b7b1a962a4a83a1"
server_key_version: "encryptionkey@0"
I20260809 22:37:37.430274 4858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:37.456646 4858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:37.462881 4858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:37.463326 4858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:37.464659 4858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:37.466305 4858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:37.466408 4858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:37.466557 4858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:37.466648 4858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:37.504915 4858 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:37189
I20260809 22:37:37.504983 5000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:37189 every 8 connection(s)
I20260809 22:37:37.506376 4858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:37.510025 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 4858
I20260809 22:37:37.510340 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 4163
Aug 09 22:37:37 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.67@KRBTEST.COM
Aug 09 22:37:37 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.124@KRBTEST.COM
Aug 09 22:37:37 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
Aug 09 22:37:37 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315057, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.125@KRBTEST.COM
I20260809 22:37:37.540242 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:37.510724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:36743 (local address 127.28.243.125:34161)
0809 22:37:37.511345 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:37.511352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:37.512219 (+ 867us) server_negotiation.cc:407] Connection header received
0809 22:37:37.515611 (+ 3392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:37.515621 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:37.515776 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:37.515922 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:37.518287 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.519512 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.523667 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.524616 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.524744 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:37.531784 (+ 7040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:37.531794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:37.531801 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:37.531866 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:37.536497 (+ 4631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.537533 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.537543 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.537548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.537719 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.538292 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.538298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.538301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.538710 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:37.538865 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:37.539555 (+ 690us) server_negotiation.cc:299] Negotiation successful
0809 22:37:37.539828 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":381,"threads_started":1}
I20260809 22:37:37.540473 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:37.511506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.28.243.67:41837)
0809 22:37:37.512122 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:37.512130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:37.512164 (+ 34us) server_negotiation.cc:407] Connection header received
0809 22:37:37.512241 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:37.512251 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:37.512502 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:37.512727 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:37.515231 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.516741 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.519049 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.520048 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.520181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:37.530859 (+ 10678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:37.530881 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:37.530901 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:37.530953 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:37.536087 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.537282 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.537292 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.537296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.537496 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.538058 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.538064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.538069 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.538551 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:37.538875 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:37.539721 (+ 846us) server_negotiation.cc:299] Negotiation successful
0809 22:37:37.540028 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":316,"threads_started":1}
I20260809 22:37:37.540525 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:37.513118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:35201 (local address 127.28.243.124:43895)
0809 22:37:37.518078 (+ 4960us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:37.518083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:37.518103 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:37.518165 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:37.518173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:37.518252 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:37.519045 (+ 793us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:37.519559 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.521171 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.523615 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.524334 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.524485 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:37.530701 (+ 6216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:37.530709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:37.530715 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:37.530757 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:37.534988 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.536214 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.536221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.536224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.536380 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.537063 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.537069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.537072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.537467 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:37.537623 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:37.539894 (+ 2271us) server_negotiation.cc:299] Negotiation successful
0809 22:37:37.540138 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4830,"thread_start_us":4693,"threads_started":1}
I20260809 22:37:37.541465 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:37.512739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.28.243.66:35201)
0809 22:37:37.513478 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:37.513498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:37.515099 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:37.518489 (+ 3390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:37.518499 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:37.519116 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:37.519390 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:37.519399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.521355 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.521360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:37.523383 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:37.523398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.524945 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.524953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:37.525432 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:37.526444 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:37.526485 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:37.530361 (+ 3876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:37.535205 (+ 4844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:37.535215 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:37.535237 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:37.535989 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:37.536535 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:37.536541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:37.536548 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:37.536905 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:37.537711 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:37.537723 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:37.538196 (+ 473us) client_negotiation.cc:769] Sending connection context
0809 22:37:37.538569 (+ 373us) client_negotiation.cc:240] Negotiation successful
0809 22:37:37.539151 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":239,"spinlock_wait_cycles":23424,"thread_start_us":558,"threads_started":1}
I20260809 22:37:37.541658 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:37.510079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.28.243.66:36743)
0809 22:37:37.510957 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:37.512109 (+ 1152us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:37.515093 (+ 2984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:37.516233 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:37.516246 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:37.517139 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:37.518037 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:37.518064 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.519765 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.519771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:37.523401 (+ 3630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:37.523421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.525144 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.525151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:37.525602 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:37.526684 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:37.526709 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:37.531488 (+ 4779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:37.536716 (+ 5228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:37.536721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:37.536725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:37.537351 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:37.537894 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:37.537900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:37.537903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:37.538131 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:37.538964 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:37.538969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:37.539202 (+ 233us) client_negotiation.cc:769] Sending connection context
0809 22:37:37.540646 (+ 1444us) client_negotiation.cc:240] Negotiation successful
0809 22:37:37.540982 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":235,"spinlock_wait_cycles":1920,"thread_start_us":367,"threads_started":1}
I20260809 22:37:37.542797 4295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:37.543303 5001 heartbeater.cc:347] Connected to a master server at 127.28.243.124:43895
I20260809 22:37:37.543372 5003 heartbeater.cc:347] Connected to a master server at 127.28.243.125:34161
I20260809 22:37:37.543689 5003 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:37.543687 5001 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:37.544816 5003 heartbeater.cc:511] Master 127.28.243.125:34161 requested a full tablet report, sending...
I20260809 22:37:37.546195 5001 heartbeater.cc:511] Master 127.28.243.124:43895 requested a full tablet report, sending...
I20260809 22:37:37.547592 4458 ts_manager.cc:194] Registered new tserver with Master: d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189)
I20260809 22:37:37.547842 4655 ts_manager.cc:194] Registered new tserver with Master: d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189)
I20260809 22:37:37.549012 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:37.514179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:58865 (local address 127.28.243.126:40309)
0809 22:37:37.517275 (+ 3096us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:37.517282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:37.517302 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:37.517361 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:37.517366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:37.517453 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:37.517574 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:37.520233 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.521358 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.523565 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.524468 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.524618 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:37.531714 (+ 7096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:37.531724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:37.531731 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:37.531798 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:37.536939 (+ 5141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.537798 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.537803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.537807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.537952 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:37.538474 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:37.538482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:37.538488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:37.538834 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:37.538958 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:37.541234 (+ 2276us) server_negotiation.cc:299] Negotiation successful
0809 22:37:37.548710 (+ 7476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2924,"thread_start_us":2822,"threads_started":1}
I20260809 22:37:37.549455 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:37.513762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.66:58865)
0809 22:37:37.514442 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:37.514465 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:37.515397 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:37.518252 (+ 2855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:37.518257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:37.519594 (+ 1337us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:37.520036 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:37.520049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.521533 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.521544 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:37.523404 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:37.523413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:37.524945 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:37.524953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:37.525432 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:37.526444 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:37.526486 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:37.530838 (+ 4352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:37.537100 (+ 6262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:37.537106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:37.537110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:37.537665 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:37.538102 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:37.538106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:37.538109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:37.538328 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:37.539044 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:37.539049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:37.539232 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 22:37:37.539401 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 22:37:37.539678 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":93,"spinlock_wait_cycles":11776,"thread_start_us":2708,"threads_started":1}
I20260809 22:37:37.550503 5002 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40309
I20260809 22:37:37.550663 5002 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:37.551090 5002 heartbeater.cc:511] Master 127.28.243.126:40309 requested a full tablet report, sending...
I20260809 22:37:37.552213 3815 ts_manager.cc:194] Re-registered known tserver with Master: d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189)
I20260809 22:37:37.553347 3815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:58865
I20260809 22:37:37.638468 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 4163
I20260809 22:37:37.658269 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:41837
--local_ip_for_outbound_sockets=127.28.243.67
--tserver_master_addrs=127.28.243.126:40309
--webserver_port=40865
--webserver_interface=127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab
--principal=oryx/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:39735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:37:37.886178 4847 heartbeater.cc:503] Master 127.28.243.126:40309 was elected leader, sending a full tablet report...
W20260809 22:37:38.026676 5015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:38.027070 5015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:38.027254 5015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:38.035295 5015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:38.035454 5015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:38.035655 5015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:38.046535 5015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:39735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab
--principal=oryx/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:41837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=40865
--tserver_master_addrs=127.28.243.126:40309,127.28.243.125:34161,127.28.243.124:43895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:38.047963 5015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:38.049388 5015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:38.064086 5020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:38.064210 5021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:38.064447 5023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:38.065887 5015 server_base.cc:1034] running on GCE node
I20260809 22:37:38.066588 5015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:38.067649 5015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:38.068881 5015 hybrid_clock.cc:648] HybridClock initialized: now 1786315058068828 us; error 34 us; skew 500 ppm
I20260809 22:37:38.069267 5015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:38 dist-test-slave-rgb1 krb5kdc[3765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315058, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:38.072999 5015 init.cc:377] Logged in from keytab as oryx/127.28.243.67@KRBTEST.COM (short username oryx)
I20260809 22:37:38.074731 5015 webserver.cc:460] Webserver started at http://127.28.243.67:40865/ using document root <none> and password file <none>
I20260809 22:37:38.075519 5015 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:38.075721 5015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:38.082062 5015 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 22:37:38.085630 5030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:38.087404 5015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:37:38.087617 5015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "9d5636127bbd471481f612cfc8b2cc63"
format_stamp: "Formatted at 2026-08-09 22:37:34 on dist-test-slave-rgb1"
server_key: "a01e84acbc187b92fa205cf3eeb74cee"
server_key_iv: "26018ea51ecb8e4edff28f6c851d3e3f"
server_key_version: "encryptionkey@0"
I20260809 22:37:38.088770 5015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:38.120007 5015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:38.125658 5015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:38.125993 5015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:38.127202 5015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:38.128736 5015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:38.128866 5015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:38.129004 5015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:38.129094 5015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:38.172084 5015 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:41837
I20260809 22:37:38.172185 5148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:41837 every 8 connection(s)
I20260809 22:37:38.173736 5015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:38.178480 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 5015
Aug 09 22:37:38 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315058, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.124@KRBTEST.COM
Aug 09 22:37:38 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315058, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.125@KRBTEST.COM
Aug 09 22:37:38 dist-test-slave-rgb1 krb5kdc[3765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315058, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:38.214401 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.177836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.28.243.67:44963)
0809 22:37:38.178798 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.178843 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.182357 (+ 3514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.183823 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.183840 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.184895 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:38.185965 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.185997 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.188228 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.188237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.191143 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.191166 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.193014 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.193025 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.193480 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.194488 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:38.194531 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:38.199037 (+ 4506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:38.204127 (+ 5090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:38.204142 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:38.204170 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:38.205118 (+ 948us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:38.205728 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:38.205734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:38.205737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:38.206062 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:38.206807 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:38.206817 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:38.207398 (+ 581us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.207740 (+ 342us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.208339 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"spinlock_wait_cycles":2816,"thread_start_us":382,"threads_started":1}
I20260809 22:37:38.218756 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.179586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:44481 (local address 127.28.243.125:34161)
0809 22:37:38.196778 (+ 17192us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.196789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.196820 (+ 31us) server_negotiation.cc:407] Connection header received
0809 22:37:38.196915 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.196923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.197036 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.197180 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.199774 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.201069 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.203042 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.203736 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.203870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.209146 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.209156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.209163 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.209232 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.213526 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.216055 (+ 2529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.216065 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.216071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.216257 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.216745 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.216753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.216757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.217157 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.217321 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.218131 (+ 810us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.218399 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14634,"thread_start_us":17745,"threads_started":1}
I20260809 22:37:38.219606 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.179424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.28.243.67:44481)
0809 22:37:38.194702 (+ 15278us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.194728 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.194880 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.197984 (+ 3104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.197992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.198921 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:38.199588 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.199608 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.201246 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.201252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.202895 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.202910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.203900 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.203908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.204088 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.205776 (+ 1688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:38.205824 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:38.208894 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:38.215234 (+ 6340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:38.215243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:38.215250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:38.215949 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:38.216378 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:38.216384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:38.216390 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:38.216612 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:38.217278 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:38.217285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:38.217484 (+ 199us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.217653 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.218081 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15075,"spinlock_wait_cycles":56832,"thread_start_us":14085,"threads_started":1}
I20260809 22:37:38.220415 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.177158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:44963 (local address 127.28.243.124:43895)
0809 22:37:38.177664 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.177669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.179028 (+ 1359us) server_negotiation.cc:407] Connection header received
0809 22:37:38.182815 (+ 3787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.182825 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.182981 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.183090 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.186241 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.187896 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.191391 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.192336 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.192469 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.199469 (+ 7000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.199478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.199486 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.199547 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.203547 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.205255 (+ 1708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.205264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.205269 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.205479 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.206236 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.206243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.206247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.206639 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.217067 (+ 10428us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.218422 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.218688 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":267,"threads_started":1}
I20260809 22:37:38.222079 5151 heartbeater.cc:347] Connected to a master server at 127.28.243.125:34161
I20260809 22:37:38.222465 5151 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:38.223510 5151 heartbeater.cc:511] Master 127.28.243.125:34161 requested a full tablet report, sending...
I20260809 22:37:38.224359 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.194192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:54415 (local address 127.28.243.126:40309)
0809 22:37:38.198276 (+ 4084us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.198316 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.200347 (+ 2031us) server_negotiation.cc:407] Connection header received
0809 22:37:38.200542 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.200550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.200676 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.200803 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.202204 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.203301 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.206883 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.207416 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.207551 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.213276 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.213284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.213290 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.213336 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.218126 (+ 4790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.220263 (+ 2137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.220272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.220278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.220461 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.222242 (+ 1781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.222247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.222250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.222750 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.222905 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.223731 (+ 826us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.223997 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3504,"thread_start_us":3398,"threads_started":1}
I20260809 22:37:38.224452 5149 heartbeater.cc:347] Connected to a master server at 127.28.243.124:43895
I20260809 22:37:38.224606 5149 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:38.225275 5149 heartbeater.cc:511] Master 127.28.243.124:43895 requested a full tablet report, sending...
I20260809 22:37:38.225888 4458 ts_manager.cc:194] Registered new tserver with Master: 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837)
I20260809 22:37:38.225934 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.194048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:54415)
0809 22:37:38.200235 (+ 6187us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.200252 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.200398 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.200824 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.200828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.201453 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:38.201813 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.201825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.203565 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.203571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.206724 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.206745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.208323 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.208333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.208663 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.209345 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:38.209366 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:38.213047 (+ 3681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:38.218287 (+ 5240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:38.218292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:38.218296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:38.218843 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:38.220898 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:38.220904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:38.220907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:38.221122 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:38.222949 (+ 1827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:38.222958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:38.223282 (+ 324us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.223483 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.223971 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6064,"thread_start_us":5891,"threads_started":1}
I20260809 22:37:38.227312 4655 ts_manager.cc:194] Registered new tserver with Master: 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837)
I20260809 22:37:38.228682 5150 heartbeater.cc:347] Connected to a master server at 127.28.243.126:40309
I20260809 22:37:38.228812 5150 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:38.229707 5150 heartbeater.cc:511] Master 127.28.243.126:40309 requested a full tablet report, sending...
I20260809 22:37:38.231550 3815 ts_manager.cc:194] Re-registered known tserver with Master: 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837)
I20260809 22:37:38.233174 3815 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:54415
I20260809 22:37:38.246521 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.203895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.28.243.126:40309)
0809 22:37:38.205585 (+ 1690us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.205591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.207964 (+ 2373us) server_negotiation.cc:407] Connection header received
0809 22:37:38.210343 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.210352 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.210454 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.210542 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.213581 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.215033 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.219031 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.219920 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.220053 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.233989 (+ 13936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.234000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.234009 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.234068 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.240093 (+ 6025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.242141 (+ 2048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.242152 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.242158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.242394 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.244334 (+ 1940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.244340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.244343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.244802 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.245088 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.245815 (+ 727us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.246042 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1545,"thread_start_us":1665,"threads_started":1}
I20260809 22:37:38.262228 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.209483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.28.243.125:34161)
0809 22:37:38.219942 (+ 10459us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.219949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.219967 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:38.220019 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.220024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.220116 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.220866 (+ 750us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.230195 (+ 9329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.232014 (+ 1819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.242687 (+ 10673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.243890 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.244052 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.250873 (+ 6821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.250879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.250882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.250920 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.255776 (+ 4856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.258496 (+ 2720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.258511 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.258515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.258792 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.259453 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.259483 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.259488 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.259955 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.260289 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.261519 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.261817 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10351,"thread_start_us":34393,"threads_started":1}
I20260809 22:37:38.252399 3815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 22:37:38.268011 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.209795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.28.243.124:43895)
0809 22:37:38.220791 (+ 10996us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.220799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.244007 (+ 23208us) server_negotiation.cc:407] Connection header received
0809 22:37:38.244351 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.244357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.244514 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.244676 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.247769 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.249276 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.251828 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.252757 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.252944 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.258082 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.258090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.258095 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.258166 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.263863 (+ 5697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.264861 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.264867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.264871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.265173 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.265807 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.265813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.265816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.266298 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.266673 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.267230 (+ 557us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.267503 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10835,"thread_start_us":33744,"threads_started":1}
W20260809 22:37:38.269075 3815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:38.314476 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.300338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.28.243.66:37189)
0809 22:37:38.301048 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.301058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.303385 (+ 2327us) server_negotiation.cc:407] Connection header received
0809 22:37:38.303710 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.303717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.303941 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.304161 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:38.305478 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.307960 (+ 2482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.311419 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.313099 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.313264 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.313587 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.313782 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.314007 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":420,"threads_started":1}
I20260809 22:37:38.314708 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.300120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:37189 (local address 127.0.0.1:43636)
0809 22:37:38.303236 (+ 3116us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.303250 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.303446 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.304141 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.304146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.304171 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:38.305041 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.305067 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.308282 (+ 3215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.308288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.311278 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.311288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.313354 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.313361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.313561 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.313600 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.313719 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.313867 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2919,"mutex_wait_us":871,"thread_start_us":883,"threads_started":1}
I20260809 22:37:38.320117 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.294982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.28.243.65:32887)
0809 22:37:38.308739 (+ 13757us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.308748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.308780 (+ 32us) server_negotiation.cc:407] Connection header received
0809 22:37:38.308911 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.308919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.309155 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.309431 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:38.311188 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.313192 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.316461 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.318941 (+ 2480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.319084 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.319304 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.319491 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.319668 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12026,"thread_start_us":11838,"threads_started":1}
I20260809 22:37:38.320128 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.294701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:32887 (local address 127.0.0.1:48312)
0809 22:37:38.302222 (+ 7521us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.302243 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.302431 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.309424 (+ 6993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.309429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.309466 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:38.310859 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.310873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.313382 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.313388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.316216 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.316232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.319160 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.319166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.319306 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.319331 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.319426 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.319558 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5185,"spinlock_wait_cycles":27520,"thread_start_us":4840,"threads_started":1}
I20260809 22:37:38.319854 4931 tablet_service.cc:1467] Processing CreateTablet for tablet 06160860c9c2481aa2ceee33bcb9ceff (DEFAULT_TABLE table=test-table [id=781d210e6603458f88cbf1c0930af7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:38.321798 4931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06160860c9c2481aa2ceee33bcb9ceff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:38.321825 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.295477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:41837 (local address 127.0.0.1:56860)
0809 22:37:38.302222 (+ 6745us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.302243 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.302422 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.306125 (+ 3703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.306130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.306156 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:38.307059 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.307075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.309829 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.309834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.315268 (+ 5434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.315284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.320759 (+ 5475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.320769 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.321015 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.321046 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.321177 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.321332 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6619,"thread_start_us":7402,"threads_started":1}
I20260809 22:37:38.322242 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.295835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.28.243.67:41837)
0809 22:37:38.305583 (+ 9748us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.305590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.305639 (+ 49us) server_negotiation.cc:407] Connection header received
0809 22:37:38.305717 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.305722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.305926 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.306142 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:38.307277 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.309696 (+ 2419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.315503 (+ 5807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.317203 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.317348 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.317587 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.321245 (+ 3658us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.321529 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9542,"thread_start_us":9206,"threads_started":1}
I20260809 22:37:38.324914 4777 tablet_service.cc:1467] Processing CreateTablet for tablet 06160860c9c2481aa2ceee33bcb9ceff (DEFAULT_TABLE table=test-table [id=781d210e6603458f88cbf1c0930af7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:38.326207 5079 tablet_service.cc:1467] Processing CreateTablet for tablet 06160860c9c2481aa2ceee33bcb9ceff (DEFAULT_TABLE table=test-table [id=781d210e6603458f88cbf1c0930af7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:38.326870 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06160860c9c2481aa2ceee33bcb9ceff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:38.327744 5079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06160860c9c2481aa2ceee33bcb9ceff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:38.350566 5183 tablet_bootstrap.cc:492] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Bootstrap starting.
I20260809 22:37:38.352993 5185 tablet_bootstrap.cc:492] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Bootstrap starting.
I20260809 22:37:38.357297 5183 tablet_bootstrap.cc:654] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:38.360275 5184 tablet_bootstrap.cc:492] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Bootstrap starting.
I20260809 22:37:38.360922 5183 log.cc:824] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:38.363293 5185 tablet_bootstrap.cc:654] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:38.364931 5185 log.cc:824] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:38.365909 5183 tablet_bootstrap.cc:492] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: No bootstrap required, opened a new log
I20260809 22:37:38.366274 5183 ts_tablet_manager.cc:1397] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260809 22:37:38.367982 5184 tablet_bootstrap.cc:654] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:38.368507 5185 tablet_bootstrap.cc:492] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: No bootstrap required, opened a new log
I20260809 22:37:38.368940 5185 ts_tablet_manager.cc:1397] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260809 22:37:38.369541 5184 log.cc:824] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:38.373693 5184 tablet_bootstrap.cc:492] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: No bootstrap required, opened a new log
I20260809 22:37:38.374061 5184 ts_tablet_manager.cc:1397] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260809 22:37:38.376932 5183 raft_consensus.cc:348] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.377473 5183 raft_consensus.cc:374] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:38.377590 5183 raft_consensus.cc:729] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d653a7f0980d485ebcb45405102b537c, State: Initialized, Role: FOLLOWER
I20260809 22:37:38.378322 5183 consensus_queue.cc:260] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.380430 5183 ts_tablet_manager.cc:1428] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260809 22:37:38.380774 5002 heartbeater.cc:503] Master 127.28.243.126:40309 was elected leader, sending a full tablet report...
I20260809 22:37:38.381382 5185 raft_consensus.cc:348] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.381893 5185 raft_consensus.cc:374] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:38.382004 5185 raft_consensus.cc:729] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef923bb58564adc9eadd9c9e9cf1346, State: Initialized, Role: FOLLOWER
I20260809 22:37:38.382678 5185 consensus_queue.cc:260] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.388116 5184 raft_consensus.cc:348] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.388886 5184 raft_consensus.cc:374] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:38.389045 5184 raft_consensus.cc:729] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5636127bbd471481f612cfc8b2cc63, State: Initialized, Role: FOLLOWER
I20260809 22:37:38.389907 5184 consensus_queue.cc:260] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.390543 5185 ts_tablet_manager.cc:1428] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260809 22:37:38.391959 5150 heartbeater.cc:503] Master 127.28.243.126:40309 was elected leader, sending a full tablet report...
I20260809 22:37:38.392511 5184 ts_tablet_manager.cc:1428] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260809 22:37:38.414456 5190 raft_consensus.cc:482] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:38.414898 5190 raft_consensus.cc:504] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.416813 5190 leader_election.cc:288] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837), d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189)
I20260809 22:37:38.424928 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.417361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:53567 (local address 127.28.243.66:37189)
0809 22:37:38.417624 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.417630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.418051 (+ 421us) server_negotiation.cc:407] Connection header received
0809 22:37:38.418308 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.418315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.418418 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.418563 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:38.419267 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.420402 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.422589 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.423815 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.423949 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.424005 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.424508 (+ 503us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.424671 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 22:37:38.425230 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.417120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:37189 (local address 127.28.243.65:53567)
0809 22:37:38.417895 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.417915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.418108 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.418616 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.418625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.418653 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:38.419092 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.419107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.420636 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.420642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.422401 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.422414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.424071 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.424079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.424278 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.424320 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.424489 (+ 169us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.424685 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":363,"threads_started":1}
I20260809 22:37:38.425812 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.417365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:47381 (local address 127.28.243.67:41837)
0809 22:37:38.417614 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.417620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.418968 (+ 1348us) server_negotiation.cc:407] Connection header received
0809 22:37:38.419130 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.419136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.419232 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.419312 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:38.420068 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.421162 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.423400 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.424751 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.424959 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.425036 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.425429 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.425565 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:37:38.425926 5193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.417956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:41837 (local address 127.28.243.65:47381)
0809 22:37:38.418827 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:38.418852 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:38.418995 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:38.419401 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:38.419406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:38.419434 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:38.419844 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.419861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.421349 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.421353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.423214 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:38.423229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.425001 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.425009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:38.425207 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.425245 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 22:37:38.425400 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 22:37:38.425545 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":1,"thread_start_us":554,"threads_started":1}
I20260809 22:37:38.426503 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06160860c9c2481aa2ceee33bcb9ceff" candidate_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d653a7f0980d485ebcb45405102b537c" is_pre_election: true
I20260809 22:37:38.426976 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06160860c9c2481aa2ceee33bcb9ceff" candidate_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5636127bbd471481f612cfc8b2cc63" is_pre_election: true
I20260809 22:37:38.427111 4953 raft_consensus.cc:2454] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef923bb58564adc9eadd9c9e9cf1346 in term 0.
I20260809 22:37:38.427665 5101 raft_consensus.cc:2454] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef923bb58564adc9eadd9c9e9cf1346 in term 0.
W20260809 22:37:38.427774 5152 tablet.cc:2369] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:38.428128 4728 leader_election.cc:302] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef923bb58564adc9eadd9c9e9cf1346, d653a7f0980d485ebcb45405102b537c; no voters:
I20260809 22:37:38.428692 5190 raft_consensus.cc:2781] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:38.428874 5190 raft_consensus.cc:482] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:38.428993 5190 raft_consensus.cc:3037] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:38.431766 5190 raft_consensus.cc:504] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.432511 5190 leader_election.cc:288] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [CANDIDATE]: Term 1 election: Requested vote from peers 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837), d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189)
I20260809 22:37:38.433164 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06160860c9c2481aa2ceee33bcb9ceff" candidate_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d653a7f0980d485ebcb45405102b537c"
I20260809 22:37:38.433210 5101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06160860c9c2481aa2ceee33bcb9ceff" candidate_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d5636127bbd471481f612cfc8b2cc63"
I20260809 22:37:38.433446 4953 raft_consensus.cc:3037] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:38.433437 5101 raft_consensus.cc:3037] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:38.436412 5101 raft_consensus.cc:2454] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef923bb58564adc9eadd9c9e9cf1346 in term 1.
I20260809 22:37:38.436774 4953 raft_consensus.cc:2454] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef923bb58564adc9eadd9c9e9cf1346 in term 1.
I20260809 22:37:38.436936 4730 leader_election.cc:302] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef923bb58564adc9eadd9c9e9cf1346, 9d5636127bbd471481f612cfc8b2cc63; no voters:
I20260809 22:37:38.437325 5190 raft_consensus.cc:2781] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:38.437803 5190 raft_consensus.cc:686] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 LEADER]: Becoming Leader. State: Replica: 6ef923bb58564adc9eadd9c9e9cf1346, State: Running, Role: LEADER
I20260809 22:37:38.438752 5190 consensus_queue.cc:237] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } }
I20260809 22:37:38.444370 3814 catalog_manager.cc:5495] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } health_report { overall_health: UNKNOWN } } }
I20260809 22:37:38.484200 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.478084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.28.243.65:32887)
0809 22:37:38.478277 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.478282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.478300 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:38.478621 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.478629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.478776 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.478993 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:38.479509 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.480346 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.481834 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.482253 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.482358 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.482994 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:38.483555 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:38.483677 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.483781 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.483855 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 22:37:38.493568 5101 raft_consensus.cc:1261] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 1 FOLLOWER]: Refusing update from remote peer 6ef923bb58564adc9eadd9c9e9cf1346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:38.493543 4953 raft_consensus.cc:1261] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Refusing update from remote peer 6ef923bb58564adc9eadd9c9e9cf1346: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:38.494696 5194 consensus_queue.cc:1035] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:38.495102 5190 consensus_queue.cc:1035] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 22:37:38.510795 5004 tablet.cc:2369] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:38.523245 5200 mvcc.cc:205] Tried to move back new op lower bound from 7316746479576215552 to 7316746479368826880. Current Snapshot: MvccSnapshot[applied={T|T < 7316746479576215552}]
I20260809 22:37:38.547705 3814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:38.548151 3814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:38.560165 3814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09d0a9690b0d4e1d987e0e8cfab5b333: Sending DeleteTablet for 3 replicas of tablet 06160860c9c2481aa2ceee33bcb9ceff
I20260809 22:37:38.561913 4931 tablet_service.cc:1514] Processing DeleteTablet for tablet 06160860c9c2481aa2ceee33bcb9ceff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:38 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:43636
I20260809 22:37:38.562252 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet 06160860c9c2481aa2ceee33bcb9ceff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:38 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:48312
I20260809 22:37:38.565658 5079 tablet_service.cc:1514] Processing DeleteTablet for tablet 06160860c9c2481aa2ceee33bcb9ceff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:38 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:56860
I20260809 22:37:38.566401 5078 tablet_service.cc:1514] Processing DeleteTablet for tablet 06160860c9c2481aa2ceee33bcb9ceff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:38 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:56860
I20260809 22:37:38.594340 5211 tablet_replica.cc:330] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: stopping tablet replica
I20260809 22:37:38.594992 5209 tablet_replica.cc:330] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: stopping tablet replica
I20260809 22:37:38.595542 5211 raft_consensus.cc:2229] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:38.596602 5211 raft_consensus.cc:2258] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:38.597324 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 3779
I20260809 22:37:38.599104 3846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:38.599804 5211 ts_tablet_manager.cc:1905] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:38.596602 5209 raft_consensus.cc:2229] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:38.603844 5209 raft_consensus.cc:2258] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:38.606796 5209 ts_tablet_manager.cc:1905] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:38.607470 5208 tablet_replica.cc:330] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: stopping tablet replica
I20260809 22:37:38.607961 5208 raft_consensus.cc:2229] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:38.608707 5211 ts_tablet_manager.cc:1918] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:38.608953 5211 log.cc:1196] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/06160860c9c2481aa2ceee33bcb9ceff
I20260809 22:37:38.609431 5208 raft_consensus.cc:2258] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:38.609819 5211 ts_tablet_manager.cc:1939] T 06160860c9c2481aa2ceee33bcb9ceff P 9d5636127bbd471481f612cfc8b2cc63: Deleting consensus metadata
I20260809 22:37:38.614147 5208 ts_tablet_manager.cc:1905] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:38.614218 5209 ts_tablet_manager.cc:1918] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:38.614384 5209 log.cc:1196] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/06160860c9c2481aa2ceee33bcb9ceff
I20260809 22:37:38.614977 5209 ts_tablet_manager.cc:1939] T 06160860c9c2481aa2ceee33bcb9ceff P 6ef923bb58564adc9eadd9c9e9cf1346: Deleting consensus metadata
I20260809 22:37:38.620056 5208 ts_tablet_manager.cc:1918] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:38.620229 5208 log.cc:1196] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/06160860c9c2481aa2ceee33bcb9ceff
I20260809 22:37:38.621249 5208 ts_tablet_manager.cc:1939] T 06160860c9c2481aa2ceee33bcb9ceff P d653a7f0980d485ebcb45405102b537c: Deleting consensus metadata
I20260809 22:37:38.631844 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.567208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.28.243.125:34161)
0809 22:37:38.567561 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.567570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.594672 (+ 27102us) server_negotiation.cc:407] Connection header received
0809 22:37:38.595402 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.595412 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.595518 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.595604 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.604672 (+ 9068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.605848 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.610459 (+ 4611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.611111 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.611223 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.622834 (+ 11611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.622845 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.622852 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.622906 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.627098 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.628140 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.628146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.628150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.628329 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.628933 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.628938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.628942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.629480 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.629765 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.631018 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.631261 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260809 22:37:38.651422 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.594957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.28.243.124:43895)
0809 22:37:38.595148 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.595154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.632341 (+ 37187us) server_negotiation.cc:407] Connection header received
0809 22:37:38.632669 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.632675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.632769 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.632847 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.636458 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.637799 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.639813 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.640535 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.640695 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.644482 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.644490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.644495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.644549 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.648149 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.649077 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.649082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.649086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.649268 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.649704 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.649709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.649712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.650062 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.650321 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.650732 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.651009 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 22:37:38.743736 3797 catalog_manager.cc:4841] TS 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837): tablet 06160860c9c2481aa2ceee33bcb9ceff (table test-table [id=781d210e6603458f88cbf1c0930af7ea]) successfully deleted
I20260809 22:37:38.743826 3795 catalog_manager.cc:4841] TS d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189): tablet 06160860c9c2481aa2ceee33bcb9ceff (table test-table [id=781d210e6603458f88cbf1c0930af7ea]) successfully deleted
I20260809 22:37:38.744131 3795 catalog_manager.cc:4841] TS 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887): tablet 06160860c9c2481aa2ceee33bcb9ceff (table test-table [id=781d210e6603458f88cbf1c0930af7ea]) successfully deleted
W20260809 22:37:38.744338 3797 catalog_manager.cc:4808] TS 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837): delete failed for tablet 06160860c9c2481aa2ceee33bcb9ceff because the tablet was not found. No further retry: Not found: Tablet not found: 06160860c9c2481aa2ceee33bcb9ceff
I20260809 22:37:38.744712 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 3779
I20260809 22:37:38.799925 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.783023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.28.243.125:34161)
0809 22:37:38.783219 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.783225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.783325 (+ 100us) server_negotiation.cc:407] Connection header received
0809 22:37:38.783654 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.783662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.783751 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.783832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.785864 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.786637 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.788649 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.789233 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.789399 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.793357 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.793366 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.793394 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.793450 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.796996 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.797719 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.797727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.797731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.797897 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.798284 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.798291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.798296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.798693 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.798989 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.799316 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.799553 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:37:38.801987 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:38.783872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.28.243.124:43895)
0809 22:37:38.784130 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:38.784136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:38.784203 (+ 67us) server_negotiation.cc:407] Connection header received
0809 22:37:38.784404 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:38.784410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:38.784491 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:38.784618 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:38.786717 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.787466 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.788900 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:38.789628 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:38.789761 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:38.794266 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:38.794273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:38.794278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:38.794322 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:38.798074 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.798954 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.798960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.798963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.799119 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:38.799566 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:38.799571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:38.799575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:38.799937 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:38.800748 (+ 811us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:38.801392 (+ 644us) server_negotiation.cc:299] Negotiation successful
0809 22:37:38.801652 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 22:37:39.527667 5219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:39.526913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.65:37653)
0809 22:37:39.527375 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:39.527501 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
W20260809 22:37:39.528265 4847 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:40309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:39.568544 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:39.567565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:53685)
0809 22:37:39.568185 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:39.568354 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":301,"threads_started":1}
W20260809 22:37:39.569003 5150 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:40309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:39.577417 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:39.576527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.66:55257)
0809 22:37:39.577091 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:39.577251 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":282,"threads_started":1}
W20260809 22:37:39.577915 5002 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:40309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:40.062713 5223 raft_consensus.cc:482] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09d0a9690b0d4e1d987e0e8cfab5b333)
I20260809 22:37:40.063063 5223 raft_consensus.cc:504] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } }
I20260809 22:37:40.064581 5223 leader_election.cc:288] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309), 93540fba9c934a9d8c4e79dc9fff7fb7 (127.28.243.125:34161)
I20260809 22:37:40.065925 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.064691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54452)
0809 22:37:40.065280 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.065543 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":256,"threads_started":1}
W20260809 22:37:40.066231 4638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:40.067876 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.067513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54468)
0809 22:37:40.067675 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.067758 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 22:37:40.068071 4638 leader_election.cc:334] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309): Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:40.070025 5228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09d0a9690b0d4e1d987e0e8cfab5b333)
I20260809 22:37:40.070271 5228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } }
I20260809 22:37:40.071194 5228 leader_election.cc:288] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309), e87b4c7179b740b68a5758353cd69f96 (127.28.243.124:43895)
I20260809 22:37:40.071831 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.065383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:34161 (local address 127.0.0.1:39820)
0809 22:37:40.065959 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.065982 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:40.066136 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:40.066562 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:40.066566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:40.066591 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:40.066981 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.066991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.068110 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.068116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.069701 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.069715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.071002 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.071008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.071188 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.071213 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 22:37:40.071325 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 22:37:40.071467 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":288,"threads_started":1}
I20260809 22:37:40.071949 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.065543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.28.243.125:34161)
0809 22:37:40.066231 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.066239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.066259 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:40.066333 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.066337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.066425 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.066539 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:40.067156 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.067892 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.069901 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.070807 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.071017 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.071098 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.071364 (+ 266us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.071506 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":364,"threads_started":1}
I20260809 22:37:40.072770 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e87b4c7179b740b68a5758353cd69f96" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" is_pre_election: true
I20260809 22:37:40.073202 4468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87b4c7179b740b68a5758353cd69f96 in term 1.
I20260809 22:37:40.073510 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.071395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54474)
0809 22:37:40.072495 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.072765 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
I20260809 22:37:40.073830 4638 leader_election.cc:302] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93540fba9c934a9d8c4e79dc9fff7fb7, e87b4c7179b740b68a5758353cd69f96; no voters: 09d0a9690b0d4e1d987e0e8cfab5b333
W20260809 22:37:40.073959 4443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:40.074359 5223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 22:37:40.074625 5223 raft_consensus.cc:482] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09d0a9690b0d4e1d987e0e8cfab5b333)
I20260809 22:37:40.074803 5223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:37:40.076031 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.075624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54490)
0809 22:37:40.075786 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.075886 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 22:37:40.076210 4443 leader_election.cc:334] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309): Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:40.078164 5223 raft_consensus.cc:504] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } }
I20260809 22:37:40.079025 5223 leader_election.cc:288] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [CANDIDATE]: Term 2 election: Requested vote from peers 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309), 93540fba9c934a9d8c4e79dc9fff7fb7 (127.28.243.125:34161)
I20260809 22:37:40.079703 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.079101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54492)
0809 22:37:40.079411 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.079538 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260809 22:37:40.079759 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e87b4c7179b740b68a5758353cd69f96" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7"
I20260809 22:37:40.080013 4468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:37:40.080957 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.080448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54498)
0809 22:37:40.080699 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.080799 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260809 22:37:40.081238 4638 leader_election.cc:334] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309): Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
I20260809 22:37:40.081607 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.072101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:43895 (local address 127.0.0.1:57244)
0809 22:37:40.072841 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.072860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:40.072983 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:40.073695 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:40.073699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:40.073727 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:40.074282 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.074297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.076204 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.076210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.078433 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.078450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.080413 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.080420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.080814 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.080851 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 22:37:40.081039 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 22:37:40.081224 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":568,"threads_started":1}
I20260809 22:37:40.081602 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.072510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.28.243.124:43895)
0809 22:37:40.073261 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.073266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.073285 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:37:40.073352 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.073359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.073430 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.073540 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:40.074553 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.075952 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.078662 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.080213 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.080410 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.080485 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.081063 (+ 578us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.081209 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":453,"threads_started":1}
I20260809 22:37:40.082993 4468 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87b4c7179b740b68a5758353cd69f96 in term 2.
I20260809 22:37:40.083465 4638 leader_election.cc:302] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93540fba9c934a9d8c4e79dc9fff7fb7, e87b4c7179b740b68a5758353cd69f96; no voters: 09d0a9690b0d4e1d987e0e8cfab5b333
I20260809 22:37:40.083804 5223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:37:40.084026 4668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e87b4c7179b740b68a5758353cd69f96" is_pre_election: true
I20260809 22:37:40.084409 5223 raft_consensus.cc:686] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [term 2 LEADER]: Becoming Leader. State: Replica: e87b4c7179b740b68a5758353cd69f96, State: Running, Role: LEADER
I20260809 22:37:40.084836 5223 consensus_queue.cc:237] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } }
I20260809 22:37:40.087433 5233 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e87b4c7179b740b68a5758353cd69f96. Latest consensus state: current_term: 2 leader_uuid: "e87b4c7179b740b68a5758353cd69f96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:40.087911 5233 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:40.087988 4443 leader_election.cc:302] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 93540fba9c934a9d8c4e79dc9fff7fb7; no voters: 09d0a9690b0d4e1d987e0e8cfab5b333, e87b4c7179b740b68a5758353cd69f96
I20260809 22:37:40.088370 5228 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 22:37:40.088873 5235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:40.093709 5235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:40.094741 5235 catalog_manager.cc:1259] Loaded cluster ID: 92f5287dfbb847b7942995cc5466fb95
I20260809 22:37:40.094821 5235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:40.095957 5235 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:40.096819 5235 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Loaded TSK: 0
I20260809 22:37:40.097730 5235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:37:40.109659 4655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:37:40.117471 4655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:40.122008 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.121533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54520)
0809 22:37:40.121735 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.121833 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 22:37:40.122104 4468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [term 2 FOLLOWER]: Refusing update from remote peer e87b4c7179b740b68a5758353cd69f96: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 22:37:40.123133 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.122708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54530)
0809 22:37:40.122887 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.122963 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 22:37:40.123224 5233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 22:37:40.123523 4638 consensus_peers.cc:487] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 -> Peer 09d0a9690b0d4e1d987e0e8cfab5b333 (127.28.243.126:40309): Couldn't send request to peer 09d0a9690b0d4e1d987e0e8cfab5b333. Status: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 22:37:40.126878 5228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e87b4c7179b740b68a5758353cd69f96. Latest consensus state: current_term: 2 leader_uuid: "e87b4c7179b740b68a5758353cd69f96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:40.127211 5228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:37:40.128513 5223 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e87b4c7179b740b68a5758353cd69f96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:40.128983 5223 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:40.129457 5228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e87b4c7179b740b68a5758353cd69f96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "09d0a9690b0d4e1d987e0e8cfab5b333" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40309 } } peers { permanent_uuid: "93540fba9c934a9d8c4e79dc9fff7fb7" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 34161 } attrs { promote: false } } peers { permanent_uuid: "e87b4c7179b740b68a5758353cd69f96" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 43895 } attrs { promote: false } } }
I20260809 22:37:40.129755 5228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93540fba9c934a9d8c4e79dc9fff7fb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:37:40.156839 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.140224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.28.243.67:41837)
0809 22:37:40.149089 (+ 8865us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.149095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.149114 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:37:40.149195 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.149198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.149329 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.150220 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:40.150434 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.151609 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.154338 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.155598 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.155736 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.155860 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.156297 (+ 437us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.156466 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8735,"thread_start_us":8626,"threads_started":1}
I20260809 22:37:40.156868 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.139762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:41837 (local address 127.0.0.1:56868)
0809 22:37:40.140328 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.140355 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:40.140467 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:40.149542 (+ 9075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:40.149550 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:40.149587 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:40.150038 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.150052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.151757 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.151765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.154158 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.154173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.155894 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.155901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.156080 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.156117 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 22:37:40.156232 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 22:37:40.156372 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":632,"threads_started":1}
I20260809 22:37:40.157811 5079 tablet_service.cc:1467] Processing CreateTablet for tablet afb4569afe9842cfaf21c13731406b10 (DEFAULT_TABLE table=test-table [id=01d09fd8ec1642bcb780f9be0661d849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:40.158396 5079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4569afe9842cfaf21c13731406b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:40.159652 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.139133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.28.243.66:37189)
0809 22:37:40.153125 (+ 13992us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.153133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.153153 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:40.153220 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.153226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.153296 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.153396 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:40.153891 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.154885 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.157328 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.158688 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.158843 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.158949 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.159222 (+ 273us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.159362 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13863,"thread_start_us":13750,"threads_started":1}
I20260809 22:37:40.159680 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.138934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:37189 (local address 127.0.0.1:43646)
0809 22:37:40.139147 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.139168 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:40.139284 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:40.153444 (+ 14160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:40.153449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:40.153477 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:40.153729 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.153740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.155061 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.155067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.157134 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.157150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.158858 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.158865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.159061 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.159085 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 22:37:40.159195 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 22:37:40.159333 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":42}
I20260809 22:37:40.160386 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.139369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:32887 (local address 127.0.0.1:48326)
0809 22:37:40.139595 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.139623 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:40.139732 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:40.152660 (+ 12928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:40.152671 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:40.152720 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:40.153123 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.153137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.155420 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.155424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.157707 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.157721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.159627 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.159634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.159825 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.159860 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 22:37:40.159943 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 22:37:40.160115 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":28}
I20260809 22:37:40.161360 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.139891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.28.243.65:32887)
0809 22:37:40.152201 (+ 12310us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.152208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.152230 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:37:40.152306 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.152311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.152405 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.152597 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:40.153299 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.154372 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.157884 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.159367 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.160657 (+ 1290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.160756 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.160810 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.160970 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12161,"thread_start_us":12007,"threads_started":1}
I20260809 22:37:40.161309 4931 tablet_service.cc:1467] Processing CreateTablet for tablet afb4569afe9842cfaf21c13731406b10 (DEFAULT_TABLE table=test-table [id=01d09fd8ec1642bcb780f9be0661d849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:40.161783 4931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4569afe9842cfaf21c13731406b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:40.162151 4777 tablet_service.cc:1467] Processing CreateTablet for tablet afb4569afe9842cfaf21c13731406b10 (DEFAULT_TABLE table=test-table [id=01d09fd8ec1642bcb780f9be0661d849]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:40.162647 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afb4569afe9842cfaf21c13731406b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:40.165715 5245 tablet_bootstrap.cc:492] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: Bootstrap starting.
I20260809 22:37:40.167332 5246 tablet_bootstrap.cc:492] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: Bootstrap starting.
I20260809 22:37:40.167538 5245 tablet_bootstrap.cc:654] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:40.169348 5246 tablet_bootstrap.cc:654] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:40.169692 5245 tablet_bootstrap.cc:492] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: No bootstrap required, opened a new log
I20260809 22:37:40.169852 5245 ts_tablet_manager.cc:1397] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:40.170557 5245 raft_consensus.cc:348] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.170863 5245 raft_consensus.cc:374] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:40.170994 5245 raft_consensus.cc:729] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5636127bbd471481f612cfc8b2cc63, State: Initialized, Role: FOLLOWER
I20260809 22:37:40.171303 5245 consensus_queue.cc:260] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.172111 5248 tablet_bootstrap.cc:492] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: Bootstrap starting.
I20260809 22:37:40.172168 5246 tablet_bootstrap.cc:492] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: No bootstrap required, opened a new log
I20260809 22:37:40.172307 5246 ts_tablet_manager.cc:1397] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260809 22:37:40.173352 5246 raft_consensus.cc:348] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.173652 5246 raft_consensus.cc:374] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:40.173758 5246 raft_consensus.cc:729] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d653a7f0980d485ebcb45405102b537c, State: Initialized, Role: FOLLOWER
I20260809 22:37:40.173893 5248 tablet_bootstrap.cc:654] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:40.174099 5246 consensus_queue.cc:260] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.177783 5245 ts_tablet_manager.cc:1428] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260809 22:37:40.179292 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.178140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:54363)
0809 22:37:40.178789 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.179022 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"threads_started":1}
I20260809 22:37:40.182934 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.182466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:49349)
0809 22:37:40.182652 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.182755 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 22:37:40.183051 5149 heartbeater.cc:503] Master 127.28.243.124:43895 was elected leader, sending a full tablet report...
I20260809 22:37:40.185901 5248 tablet_bootstrap.cc:492] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: No bootstrap required, opened a new log
I20260809 22:37:40.186106 5248 ts_tablet_manager.cc:1397] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260809 22:37:40.187117 5248 raft_consensus.cc:348] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.187363 5248 raft_consensus.cc:374] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:40.187505 5248 raft_consensus.cc:729] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef923bb58564adc9eadd9c9e9cf1346, State: Initialized, Role: FOLLOWER
I20260809 22:37:40.187788 5248 consensus_queue.cc:260] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.188995 5246 ts_tablet_manager.cc:1428] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: Time spent starting tablet: real 0.017s user 0.005s sys 0.013s
I20260809 22:37:40.190675 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.189668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.66:56481)
0809 22:37:40.190216 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.190396 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"threads_started":1}
I20260809 22:37:40.190717 5250 raft_consensus.cc:482] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:40.190989 5250 raft_consensus.cc:504] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.194164 5001 heartbeater.cc:503] Master 127.28.243.124:43895 was elected leader, sending a full tablet report...
I20260809 22:37:40.195210 5250 leader_election.cc:288] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189), 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887)
I20260809 22:37:40.195858 5248 ts_tablet_manager.cc:1428] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260809 22:37:40.196133 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.193737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.65:60635)
0809 22:37:40.195298 (+ 1561us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.195763 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1384,"thread_start_us":1229,"threads_started":1}
I20260809 22:37:40.197284 4846 heartbeater.cc:503] Master 127.28.243.124:43895 was elected leader, sending a full tablet report...
I20260809 22:37:40.198215 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.197686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.66:49441)
0809 22:37:40.197932 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.198034 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260809 22:37:40.200136 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.198228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.65:40607)
0809 22:37:40.199848 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.199952 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1491}
I20260809 22:37:40.204234 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.195556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:33299 (local address 127.28.243.66:37189)
0809 22:37:40.196680 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.196688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.196867 (+ 179us) server_negotiation.cc:407] Connection header received
0809 22:37:40.197461 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.197468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.197547 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.197628 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:40.198490 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.199705 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.201862 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.203006 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.203225 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.203313 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.203782 (+ 469us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.203957 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949}
I20260809 22:37:40.204269 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.195528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:37189 (local address 127.28.243.67:33299)
0809 22:37:40.196764 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.196785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:40.197000 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:40.197854 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:40.197860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:40.197891 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:40.198287 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.198304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.199900 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.199905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.201683 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.201701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.203230 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.203237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.203574 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.203614 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 22:37:40.203730 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 22:37:40.203884 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084}
I20260809 22:37:40.204919 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4569afe9842cfaf21c13731406b10" candidate_uuid: "9d5636127bbd471481f612cfc8b2cc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d653a7f0980d485ebcb45405102b537c" is_pre_election: true
I20260809 22:37:40.205202 4953 raft_consensus.cc:2454] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5636127bbd471481f612cfc8b2cc63 in term 0.
I20260809 22:37:40.205444 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.196971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:39241 (local address 127.28.243.65:32887)
0809 22:37:40.197272 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.197278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.198600 (+ 1322us) server_negotiation.cc:407] Connection header received
0809 22:37:40.198872 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.198877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.198953 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.199048 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:40.199707 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.200886 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.202691 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.204108 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.204276 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.204529 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.205042 (+ 513us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.205190 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260809 22:37:40.205552 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.195903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:32887 (local address 127.28.243.67:39241)
0809 22:37:40.198490 (+ 2587us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.198511 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:40.198700 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:40.199257 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:40.199262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:40.199304 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:40.199545 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.199555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.201044 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.201050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.202487 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:40.202502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.204700 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.204705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:40.204843 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.204868 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 22:37:40.204989 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 22:37:40.205148 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2469,"thread_start_us":514,"threads_started":1}
I20260809 22:37:40.205986 5031 leader_election.cc:302] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d5636127bbd471481f612cfc8b2cc63, d653a7f0980d485ebcb45405102b537c; no voters:
I20260809 22:37:40.206455 5250 raft_consensus.cc:2781] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:40.206614 5250 raft_consensus.cc:482] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:40.206758 5250 raft_consensus.cc:3037] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:40.207340 4797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4569afe9842cfaf21c13731406b10" candidate_uuid: "9d5636127bbd471481f612cfc8b2cc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" is_pre_election: true
I20260809 22:37:40.207744 4797 raft_consensus.cc:2454] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d5636127bbd471481f612cfc8b2cc63 in term 0.
I20260809 22:37:40.209758 5250 raft_consensus.cc:504] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.210376 5250 leader_election.cc:288] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [CANDIDATE]: Term 1 election: Requested vote from peers d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189), 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887)
I20260809 22:37:40.211004 4953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4569afe9842cfaf21c13731406b10" candidate_uuid: "9d5636127bbd471481f612cfc8b2cc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d653a7f0980d485ebcb45405102b537c"
I20260809 22:37:40.211086 4797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afb4569afe9842cfaf21c13731406b10" candidate_uuid: "9d5636127bbd471481f612cfc8b2cc63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef923bb58564adc9eadd9c9e9cf1346"
I20260809 22:37:40.211270 4953 raft_consensus.cc:3037] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:40.211315 4797 raft_consensus.cc:3037] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:40.213761 4953 raft_consensus.cc:2454] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5636127bbd471481f612cfc8b2cc63 in term 1.
I20260809 22:37:40.213838 4797 raft_consensus.cc:2454] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d5636127bbd471481f612cfc8b2cc63 in term 1.
I20260809 22:37:40.214475 5031 leader_election.cc:302] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d5636127bbd471481f612cfc8b2cc63, d653a7f0980d485ebcb45405102b537c; no voters:
I20260809 22:37:40.214953 5250 raft_consensus.cc:2781] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:40.215504 5250 raft_consensus.cc:686] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 1 LEADER]: Becoming Leader. State: Replica: 9d5636127bbd471481f612cfc8b2cc63, State: Running, Role: LEADER
I20260809 22:37:40.215960 5250 consensus_queue.cc:237] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } }
I20260809 22:37:40.216621 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.216141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:60327)
0809 22:37:40.216314 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.216423 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 22:37:40.219916 4655 catalog_manager.cc:5495] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67). New cstate: current_term: 1 leader_uuid: "9d5636127bbd471481f612cfc8b2cc63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d5636127bbd471481f612cfc8b2cc63" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 41837 } health_report { overall_health: HEALTHY } } }
I20260809 22:37:40.246860 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.241281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.28.243.67:41837)
0809 22:37:40.241513 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.241521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.241542 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:37:40.241822 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.241826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.241920 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.242088 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:40.242442 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.243191 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.244605 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.245046 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.245196 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.245621 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:40.246207 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:40.246376 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.246480 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.246576 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 22:37:40.251295 4953 raft_consensus.cc:1261] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Refusing update from remote peer 9d5636127bbd471481f612cfc8b2cc63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:40.251660 4797 raft_consensus.cc:1261] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 FOLLOWER]: Refusing update from remote peer 9d5636127bbd471481f612cfc8b2cc63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:40.252215 5261 consensus_queue.cc:1035] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d653a7f0980d485ebcb45405102b537c" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 37189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:40.252779 5250 consensus_queue.cc:1035] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef923bb58564adc9eadd9c9e9cf1346" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 32887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:40.255724 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.255095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.65:60611)
0809 22:37:40.255369 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.255511 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 22:37:40.257094 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.256542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.66:46229)
0809 22:37:40.256738 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.256878 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260809 22:37:40.269026 5264 mvcc.cc:205] Tried to move back new op lower bound from 7316746486781423616 to 7316746486649503744. Current Snapshot: MvccSnapshot[applied={T|T < 7316746486781423616}]
I20260809 22:37:40.271984 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.270462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.65:53455)
0809 22:37:40.271614 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.271755 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":979}
I20260809 22:37:40.276515 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.276004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.66:55099)
0809 22:37:40.276202 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.276329 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 22:37:40.279013 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.278357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:59193)
0809 22:37:40.278694 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.278790 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260809 22:37:40.281049 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.280330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.28.243.67:40951)
0809 22:37:40.280783 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.280884 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334}
I20260809 22:37:40.288875 4655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:40.289215 4655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:40.296492 4655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e87b4c7179b740b68a5758353cd69f96: Sending DeleteTablet for 3 replicas of tablet afb4569afe9842cfaf21c13731406b10
I20260809 22:37:40.297710 4931 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4569afe9842cfaf21c13731406b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:40 UTC) from {username='oryx', principal='oryx/127.28.243.124@KRBTEST.COM'} at 127.0.0.1:43646
I20260809 22:37:40.297729 5079 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4569afe9842cfaf21c13731406b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:40 UTC) from {username='oryx', principal='oryx/127.28.243.124@KRBTEST.COM'} at 127.0.0.1:56868
I20260809 22:37:40.298272 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet afb4569afe9842cfaf21c13731406b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:40 UTC) from {username='oryx', principal='oryx/127.28.243.124@KRBTEST.COM'} at 127.0.0.1:48326
I20260809 22:37:40.298532 5277 tablet_replica.cc:330] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: stopping tablet replica
I20260809 22:37:40.298779 5277 raft_consensus.cc:2229] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:40.299129 5277 raft_consensus.cc:2258] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:40.299695 5279 tablet_replica.cc:330] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: stopping tablet replica
I20260809 22:37:40.299948 5279 raft_consensus.cc:2229] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:40.300302 5277 ts_tablet_manager.cc:1905] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:40.300323 5279 raft_consensus.cc:2258] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:40.300621 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 4641
I20260809 22:37:40.301575 5279 ts_tablet_manager.cc:1905] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:40.307296 5277 ts_tablet_manager.cc:1918] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:40.307492 5277 log.cc:1196] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/afb4569afe9842cfaf21c13731406b10
I20260809 22:37:40.308015 5277 ts_tablet_manager.cc:1939] T afb4569afe9842cfaf21c13731406b10 P d653a7f0980d485ebcb45405102b537c: Deleting consensus metadata
I20260809 22:37:40.308790 5279 ts_tablet_manager.cc:1918] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:40.308998 5279 log.cc:1196] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/afb4569afe9842cfaf21c13731406b10
I20260809 22:37:40.309530 5279 ts_tablet_manager.cc:1939] T afb4569afe9842cfaf21c13731406b10 P 6ef923bb58564adc9eadd9c9e9cf1346: Deleting consensus metadata
I20260809 22:37:40.309849 4637 catalog_manager.cc:4841] TS d653a7f0980d485ebcb45405102b537c (127.28.243.66:37189): tablet afb4569afe9842cfaf21c13731406b10 (table test-table [id=01d09fd8ec1642bcb780f9be0661d849]) successfully deleted
I20260809 22:37:40.310654 4637 catalog_manager.cc:4841] TS 6ef923bb58564adc9eadd9c9e9cf1346 (127.28.243.65:32887): tablet afb4569afe9842cfaf21c13731406b10 (table test-table [id=01d09fd8ec1642bcb780f9be0661d849]) successfully deleted
I20260809 22:37:40.313032 5278 tablet_replica.cc:330] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: stopping tablet replica
I20260809 22:37:40.313290 5278 raft_consensus.cc:2229] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:40.314035 5278 raft_consensus.cc:2258] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:40.315366 5278 ts_tablet_manager.cc:1905] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:40.322558 5278 ts_tablet_manager.cc:1918] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:40.322844 5278 log.cc:1196] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/afb4569afe9842cfaf21c13731406b10
I20260809 22:37:40.323449 5278 ts_tablet_manager.cc:1939] T afb4569afe9842cfaf21c13731406b10 P 9d5636127bbd471481f612cfc8b2cc63: Deleting consensus metadata
I20260809 22:37:40.324906 4639 catalog_manager.cc:4841] TS 9d5636127bbd471481f612cfc8b2cc63 (127.28.243.67:41837): tablet afb4569afe9842cfaf21c13731406b10 (table test-table [id=01d09fd8ec1642bcb780f9be0661d849]) successfully deleted
I20260809 22:37:40.332392 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.302253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48328 (local address 127.28.243.65:32887)
0809 22:37:40.302448 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.302458 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.308928 (+ 6470us) server_negotiation.cc:407] Connection header received
0809 22:37:40.309154 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.309160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.309256 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.309383 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:40.315532 (+ 6149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.316822 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.318935 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.319700 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.319907 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.324065 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:40.324090 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:40.324112 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:40.324165 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:40.328820 (+ 4655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:40.329685 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:40.329696 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:40.329700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:40.329923 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:40.330433 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:40.330439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:40.330443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:40.330996 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:40.331353 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.331669 (+ 316us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.331951 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 22:37:40.333276 4841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:40.465960 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 4641
I20260809 22:37:40.494062 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 4858
I20260809 22:37:40.512814 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.495235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.28.243.66:37189)
0809 22:37:40.495412 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.495416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.495434 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:40.495490 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.495494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.495565 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.495652 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:40.497675 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.498549 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.500389 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.501140 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.501265 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.504738 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:40.504759 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:40.504779 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:40.504824 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:40.509297 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:40.510232 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:40.510242 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:40.510246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:40.510413 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:40.510910 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:40.510916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:40.510920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:40.511523 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:40.511827 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.512148 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.512406 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 22:37:40.513576 4996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:40.617110 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 4858
I20260809 22:37:40.639003 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 5015
I20260809 22:37:40.654865 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.639837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.28.243.67:41837)
0809 22:37:40.640040 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:40.640047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:40.640205 (+ 158us) server_negotiation.cc:407] Connection header received
0809 22:37:40.640443 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:40.640448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:40.640544 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:40.640703 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:40.642223 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.643039 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.644283 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:40.645046 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:40.645207 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:40.647952 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:40.647968 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:40.647984 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:40.648033 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:40.651580 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:40.652463 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:40.652480 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:40.652482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:40.652706 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:40.653117 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:40.653122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:40.653126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:40.653537 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:40.653887 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:40.654222 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 22:37:40.654475 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 22:37:40.655586 5143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:40.728205 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.727398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54532)
0809 22:37:40.727928 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.728026 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":295,"threads_started":1}
I20260809 22:37:40.729760 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:40.729383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:40309 (local address 127.0.0.1:54548)
0809 22:37:40.729540 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:40.729627 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260809 22:37:40.794456 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 5015
I20260809 22:37:40.819473 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 4307
I20260809 22:37:40.820725 4490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:40.944872 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 4307
I20260809 22:37:40.969790 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 4416
I20260809 22:37:40.971022 4689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:41.100412 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 4416
2026-08-09T22:37:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10326 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:41 dist-test-slave-rgb1 krb5kdc[5291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:41 dist-test-slave-rgb1 krb5kdc[5291](info): set up 2 sockets
Aug 09 22:37:41 dist-test-slave-rgb1 krb5kdc[5291](info): commencing operation
krb5kdc: starting...
W20260809 22:37:42.322919 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:42 dist-test-slave-rgb1 krb5kdc[5291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab.
Entry for principal oryx/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab.
I20260809 22:37:42.621789 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36667
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab
--principal=oryx/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:42.971124 5304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:42.971560 5304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:42.971760 5304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:42.979163 5304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:42.979344 5304 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 22:37:42.979425 5304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:42.979496 5304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:42.979549 5304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:42.979605 5304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:42.989833 5304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:36667
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.126.keytab
--principal=oryx/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:42.991147 5304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:42.992467 5304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:43.006356 5313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:43.006412 5311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:43.006441 5310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:43.008539 5304 server_base.cc:1034] running on GCE node
I20260809 22:37:43.009595 5304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:43.011271 5304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:43.012681 5304 hybrid_clock.cc:648] HybridClock initialized: now 1786315063012578 us; error 63 us; skew 500 ppm
I20260809 22:37:43.013048 5304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:43 dist-test-slave-rgb1 krb5kdc[5291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315063, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:43.017555 5304 init.cc:377] Logged in from keytab as oryx/127.28.243.126@KRBTEST.COM (short username oryx)
I20260809 22:37:43.019150 5304 webserver.cc:460] Webserver started at http://127.28.243.126:44549/ using document root <none> and password file <none>
I20260809 22:37:43.019863 5304 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:43.019958 5304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:43.020234 5304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:43.023466 5304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "37562621482d4d3097cfa2c40c4cfba9"
format_stamp: "Formatted at 2026-08-09 22:37:43 on dist-test-slave-rgb1"
server_key: "1104bbf7bb78839864de1fb57f8c717d"
server_key_iv: "f0a0a056f2ed930063c580c32ae33eeb"
server_key_version: "encryptionkey@0"
I20260809 22:37:43.024322 5304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "37562621482d4d3097cfa2c40c4cfba9"
format_stamp: "Formatted at 2026-08-09 22:37:43 on dist-test-slave-rgb1"
server_key: "1104bbf7bb78839864de1fb57f8c717d"
server_key_iv: "f0a0a056f2ed930063c580c32ae33eeb"
server_key_version: "encryptionkey@0"
I20260809 22:37:43.029870 5304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 22:37:43.034260 5320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:43.035822 5304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:37:43.036093 5304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "37562621482d4d3097cfa2c40c4cfba9"
format_stamp: "Formatted at 2026-08-09 22:37:43 on dist-test-slave-rgb1"
server_key: "1104bbf7bb78839864de1fb57f8c717d"
server_key_iv: "f0a0a056f2ed930063c580c32ae33eeb"
server_key_version: "encryptionkey@0"
I20260809 22:37:43.036338 5304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:43.049568 5304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:43.054463 5304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:43.054795 5304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:43.080472 5304 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:36667
I20260809 22:37:43.080524 5385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:36667 every 8 connection(s)
I20260809 22:37:43.081995 5304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:43.086012 5386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:43.092188 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 5304
I20260809 22:37:43.092444 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:43.093170 29645 external_mini_cluster.cc:1428] Setting key 3b2e91dd9152a9b24ef4359f55a65b57
I20260809 22:37:43.098608 5386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Bootstrap starting.
I20260809 22:37:43.102692 5386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:43.104072 5386 log.cc:824] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:43.107851 5386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: No bootstrap required, opened a new log
Aug 09 22:37:43 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:43.118839 5386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37562621482d4d3097cfa2c40c4cfba9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36667 } }
I20260809 22:37:43.119467 5386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:43.119598 5386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37562621482d4d3097cfa2c40c4cfba9, State: Initialized, Role: FOLLOWER
I20260809 22:37:43.120343 5386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37562621482d4d3097cfa2c40c4cfba9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36667 } }
I20260809 22:37:43.120654 5386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:43.120814 5386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:43.121042 5386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:43.122115 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:43.095559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.28.243.126:36667)
0809 22:37:43.096656 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:43.096669 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:43.096710 (+ 41us) server_negotiation.cc:407] Connection header received
0809 22:37:43.098570 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:43.098597 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:43.099079 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:43.099417 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:43.101478 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.103113 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.105084 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.106089 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.106282 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:43.111914 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:43.111923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:43.111943 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:43.111996 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:43.116204 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:43.117370 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:43.117381 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:43.117390 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:43.117642 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:43.118377 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:43.118386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:43.118389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:43.119446 (+ 1057us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:43.119801 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:43.120341 (+ 540us) server_negotiation.cc:299] Negotiation successful
0809 22:37:43.120887 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":547,"threads_started":1}
I20260809 22:37:43.123721 5386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37562621482d4d3097cfa2c40c4cfba9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36667 } }
I20260809 22:37:43.124291 5386 leader_election.cc:302] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37562621482d4d3097cfa2c40c4cfba9; no voters:
I20260809 22:37:43.125360 5386 leader_election.cc:288] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:43.125434 5391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:43.126195 5391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [term 1 LEADER]: Becoming Leader. State: Replica: 37562621482d4d3097cfa2c40c4cfba9, State: Running, Role: LEADER
I20260809 22:37:43.126844 5391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37562621482d4d3097cfa2c40c4cfba9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36667 } }
I20260809 22:37:43.127779 5386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:43.130460 5394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37562621482d4d3097cfa2c40c4cfba9. Latest consensus state: current_term: 1 leader_uuid: "37562621482d4d3097cfa2c40c4cfba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37562621482d4d3097cfa2c40c4cfba9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36667 } } }
I20260809 22:37:43.130693 5393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37562621482d4d3097cfa2c40c4cfba9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37562621482d4d3097cfa2c40c4cfba9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36667 } } }
I20260809 22:37:43.135783 5394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:43.135903 5393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:43.137029 5399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:43.144548 5399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:43.157315 5399 catalog_manager.cc:1347] Generated new cluster ID: 67c6f33d69da4a61972c45ea0ff704f6
I20260809 22:37:43.157455 5399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:43.186733 5399 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:43.188213 5399 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:43.194684 5399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Generated new TSK 0
I20260809 22:37:43.195731 5399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab.
Entry for principal oryx/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab.
I20260809 22:37:43.273669 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36667
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab
--principal=oryx/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:43.634058 5414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:43.634413 5414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:43.634622 5414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:43.642630 5414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:43.642792 5414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:43.642935 5414 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:37:43.643070 5414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:43.653384 5414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab
--principal=oryx/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:43.654908 5414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:43.656380 5414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:43.671046 5419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:43.671119 5422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:43.671096 5420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:43.672252 5414 server_base.cc:1034] running on GCE node
I20260809 22:37:43.673023 5414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:43.674150 5414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:43.675472 5414 hybrid_clock.cc:648] HybridClock initialized: now 1786315063675378 us; error 61 us; skew 500 ppm
I20260809 22:37:43.675848 5414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:43 dist-test-slave-rgb1 krb5kdc[5291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315063, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:43.683606 5414 init.cc:377] Logged in from keytab as oryx/127.28.243.65@KRBTEST.COM (short username oryx)
I20260809 22:37:43.685343 5414 webserver.cc:460] Webserver started at http://127.28.243.65:44525/ using document root <none> and password file <none>
I20260809 22:37:43.686054 5414 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:43.686163 5414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:43.686445 5414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:43.689605 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "15a573f7742744958ab25e4654588469"
format_stamp: "Formatted at 2026-08-09 22:37:43 on dist-test-slave-rgb1"
server_key: "7c24b5cd85b5b6a89726b137de98b286"
server_key_iv: "46f11d6f80c883b3db3d9f3493c646ec"
server_key_version: "encryptionkey@0"
I20260809 22:37:43.690419 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "15a573f7742744958ab25e4654588469"
format_stamp: "Formatted at 2026-08-09 22:37:43 on dist-test-slave-rgb1"
server_key: "7c24b5cd85b5b6a89726b137de98b286"
server_key_iv: "46f11d6f80c883b3db3d9f3493c646ec"
server_key_version: "encryptionkey@0"
I20260809 22:37:43.695926 5414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 22:37:43.699836 5429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:43.701655 5414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:43.701823 5414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "15a573f7742744958ab25e4654588469"
format_stamp: "Formatted at 2026-08-09 22:37:43 on dist-test-slave-rgb1"
server_key: "7c24b5cd85b5b6a89726b137de98b286"
server_key_iv: "46f11d6f80c883b3db3d9f3493c646ec"
server_key_version: "encryptionkey@0"
I20260809 22:37:43.702047 5414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:43.725596 5414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:43.730710 5414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:43.730980 5414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:43.734360 5414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:43.734467 5414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:43.734656 5414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:43.734752 5414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:37:43 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315063, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:43.763684 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:43.735583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36667 (local address 127.28.243.65:48253)
0809 22:37:43.736742 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:43.736806 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:43.739968 (+ 3162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:43.741120 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:43.741133 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:43.742369 (+ 1236us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:43.743793 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:43.743832 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.745543 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.745549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:43.747901 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:43.747938 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.749137 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.749148 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:43.749350 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:43.750276 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:43.750301 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:43.753745 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:43.757810 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:43.757826 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:43.757844 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:43.758682 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:43.759219 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:43.759224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:43.759228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:43.759585 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:43.760435 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:43.760442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:43.760928 (+ 486us) client_negotiation.cc:769] Sending connection context
0809 22:37:43.761454 (+ 526us) client_negotiation.cc:240] Negotiation successful
0809 22:37:43.762186 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":533,"threads_started":1}
I20260809 22:37:43.763978 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:43.734860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:48253 (local address 127.28.243.126:36667)
0809 22:37:43.735720 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:43.735727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:43.736971 (+ 1244us) server_negotiation.cc:407] Connection header received
0809 22:37:43.740394 (+ 3423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:43.740403 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:43.740543 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:43.740691 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:43.744188 (+ 3497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.745257 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.748109 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.748801 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.748956 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:43.754048 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:43.754055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:43.754060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:43.754128 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:43.757620 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:43.758829 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:43.758837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:43.758843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:43.759053 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:43.759761 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:43.759771 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:43.759777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:43.760265 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:43.760461 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:43.763105 (+ 2644us) server_negotiation.cc:299] Negotiation successful
0809 22:37:43.763460 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":580,"threads_started":1}
I20260809 22:37:43.786585 5414 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:37089
I20260809 22:37:43.786638 5558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:37089 every 8 connection(s)
I20260809 22:37:43.788105 5414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:43.790908 5559 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36667
I20260809 22:37:43.791204 5559 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:43.792117 5559 heartbeater.cc:511] Master 127.28.243.126:36667 requested a full tablet report, sending...
I20260809 22:37:43.792981 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:43.776829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:46425 (local address 127.28.243.126:36667)
0809 22:37:43.777008 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:43.777013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:43.777343 (+ 330us) server_negotiation.cc:407] Connection header received
0809 22:37:43.777524 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:43.777535 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:43.777650 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:43.777776 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:43.779403 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.780323 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.783218 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.783824 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.783976 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:43.786605 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:43.786610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:43.786614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:43.786649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:43.789883 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:43.790890 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:43.790900 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:43.790902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:43.791032 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:43.791495 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:43.791498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:43.791500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:43.791820 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:43.791944 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:43.792478 (+ 534us) server_negotiation.cc:299] Negotiation successful
0809 22:37:43.792740 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 22:37:43.793346 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:43.776140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36667 (local address 127.28.243.65:46425)
0809 22:37:43.777215 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:43.777232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:43.777378 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:43.777859 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:43.777864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:43.778593 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:43.779219 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:43.779232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.780512 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.780525 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:43.783010 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:43.783036 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:43.784222 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:43.784227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:43.784348 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:43.785163 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:43.785186 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:43.786350 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:43.790066 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:43.790071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:43.790074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:43.790747 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:43.791173 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:43.791180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:43.791190 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:43.791383 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:43.791963 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:43.791968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:43.792193 (+ 225us) client_negotiation.cc:769] Sending connection context
0809 22:37:43.792380 (+ 187us) client_negotiation.cc:240] Negotiation successful
0809 22:37:43.792807 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"thread_start_us":605,"threads_started":1}
I20260809 22:37:43.794875 5337 ts_manager.cc:194] Registered new tserver with Master: 15a573f7742744958ab25e4654588469 (127.28.243.65:37089)
I20260809 22:37:43.795589 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 5414
I20260809 22:37:43.795799 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:43.796953 5337 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:48253
I20260809 22:37:43.796999 29645 external_mini_cluster.cc:1428] Setting key 560e9fe7af9f9c82bd0c9b1df4b298ac
WARNING: no policy specified for oryx/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab.
Entry for principal oryx/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab.
I20260809 22:37:43.861398 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36667
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab
--principal=oryx/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:44.217226 5563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:44.217523 5563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:44.217634 5563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:44.225169 5563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:44.225291 5563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:44.225378 5563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:37:44.225461 5563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:44.235642 5563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab
--principal=oryx/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:44.237015 5563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:44.238346 5563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:44.252341 5568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:44.252405 5572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:44.252383 5570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:44.253386 5563 server_base.cc:1034] running on GCE node
I20260809 22:37:44.254055 5563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:44.255296 5563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:44.256708 5563 hybrid_clock.cc:648] HybridClock initialized: now 1786315064256615 us; error 59 us; skew 500 ppm
I20260809 22:37:44.257112 5563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:44 dist-test-slave-rgb1 krb5kdc[5291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315064, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:44.261116 5563 init.cc:377] Logged in from keytab as oryx/127.28.243.66@KRBTEST.COM (short username oryx)
I20260809 22:37:44.263103 5563 webserver.cc:460] Webserver started at http://127.28.243.66:45515/ using document root <none> and password file <none>
I20260809 22:37:44.263903 5563 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:44.264056 5563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:44.264415 5563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:44.267923 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "838c84591d7d42e7b18595471c954de8"
format_stamp: "Formatted at 2026-08-09 22:37:44 on dist-test-slave-rgb1"
server_key: "c90ce6143bbb27e32e30cb58ed2d3f75"
server_key_iv: "d499382f18fc02093d0fe5c53e782bb1"
server_key_version: "encryptionkey@0"
I20260809 22:37:44.268862 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "838c84591d7d42e7b18595471c954de8"
format_stamp: "Formatted at 2026-08-09 22:37:44 on dist-test-slave-rgb1"
server_key: "c90ce6143bbb27e32e30cb58ed2d3f75"
server_key_iv: "d499382f18fc02093d0fe5c53e782bb1"
server_key_version: "encryptionkey@0"
I20260809 22:37:44.275580 5563 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:37:44.279767 5580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:44.281582 5563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260809 22:37:44.281791 5563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "838c84591d7d42e7b18595471c954de8"
format_stamp: "Formatted at 2026-08-09 22:37:44 on dist-test-slave-rgb1"
server_key: "c90ce6143bbb27e32e30cb58ed2d3f75"
server_key_iv: "d499382f18fc02093d0fe5c53e782bb1"
server_key_version: "encryptionkey@0"
I20260809 22:37:44.282052 5563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:44.306170 5563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:44.310870 5563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:44.311164 5563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:44.314196 5563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:44.314316 5563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:44.314477 5563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:44.314563 5563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:37:44 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315064, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:44.341529 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.315548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36667 (local address 127.28.243.66:42685)
0809 22:37:44.316525 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:44.316622 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:44.318687 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:44.319538 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:44.319566 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:44.320484 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:44.321442 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.321466 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.323014 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.323022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.326058 (+ 3036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.326081 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.327023 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.327034 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.327446 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.328322 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:44.328379 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:44.331922 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:44.336593 (+ 4671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.336610 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.336631 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.337232 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.337750 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.337756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.337764 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.338015 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.338623 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:44.338632 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:44.339325 (+ 693us) client_negotiation.cc:769] Sending connection context
0809 22:37:44.339696 (+ 371us) client_negotiation.cc:240] Negotiation successful
0809 22:37:44.340332 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":446,"threads_started":1}
I20260809 22:37:44.342643 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.314807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:42685 (local address 127.28.243.126:36667)
0809 22:37:44.315536 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:44.315544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:44.316753 (+ 1209us) server_negotiation.cc:407] Connection header received
0809 22:37:44.318994 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:44.319004 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:44.319120 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:44.319248 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:44.321729 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.322796 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.326286 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.326737 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.326846 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.332190 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:44.332196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:44.332199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:44.332237 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:44.336325 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.337408 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.337415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.337421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.337602 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.338170 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.338175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.338179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.338471 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:44.338691 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:44.342094 (+ 3403us) server_negotiation.cc:299] Negotiation successful
0809 22:37:44.342367 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":468,"threads_started":1}
I20260809 22:37:44.375859 5563 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:43927
I20260809 22:37:44.375885 5708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:43927 every 8 connection(s)
I20260809 22:37:44.377254 5563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:44.378643 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.360201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:49279 (local address 127.28.243.126:36667)
0809 22:37:44.360468 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:44.360478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:44.361432 (+ 954us) server_negotiation.cc:407] Connection header received
0809 22:37:44.361703 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:44.361708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:44.361780 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:44.361889 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:44.363617 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.364482 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.367438 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.368242 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.368382 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.371713 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:44.371723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:44.371732 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:44.371782 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:44.375321 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.376233 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.376246 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.376248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.376383 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.376852 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.376859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.376863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.377227 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:44.377511 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:44.377983 (+ 472us) server_negotiation.cc:299] Negotiation successful
0809 22:37:44.378240 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 22:37:44.378796 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.360212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36667 (local address 127.28.243.66:49279)
0809 22:37:44.361274 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:44.361301 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:44.361515 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:44.361968 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:44.361976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:44.362772 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:44.363268 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.363285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.364732 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.364741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.367203 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.367225 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.368857 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.368865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.369135 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.369962 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:44.369994 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:44.371426 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:44.375488 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.375494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.375499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.376087 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.376524 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.376530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.376534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.376727 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.377458 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:44.377465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:44.377685 (+ 220us) client_negotiation.cc:769] Sending connection context
0809 22:37:44.377845 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 22:37:44.378252 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":535,"threads_started":1}
I20260809 22:37:44.379698 5709 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36667
I20260809 22:37:44.380878 5709 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:44.381553 5709 heartbeater.cc:511] Master 127.28.243.126:36667 requested a full tablet report, sending...
I20260809 22:37:44.382128 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 5563
I20260809 22:37:44.382339 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:44.382908 29645 external_mini_cluster.cc:1428] Setting key e326cc3e11910dc9041ae172c707155f
I20260809 22:37:44.383426 5337 ts_manager.cc:194] Registered new tserver with Master: 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927)
I20260809 22:37:44.384479 5337 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:42685
WARNING: no policy specified for oryx/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab.
Entry for principal oryx/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab.
I20260809 22:37:44.445464 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36667
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab
--principal=oryx/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:37:44.800539 5559 heartbeater.cc:503] Master 127.28.243.126:36667 was elected leader, sending a full tablet report...
W20260809 22:37:44.820458 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:44.820822 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:44.820932 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:44.828375 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:44.828503 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:44.828665 5714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 22:37:44.828758 5714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:44.838955 5714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab
--principal=oryx/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.243.126:36667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:44.840396 5714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:44.841893 5714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:44.856336 5719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:44.856411 5720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:44.856420 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:44.857548 5714 server_base.cc:1034] running on GCE node
I20260809 22:37:44.858261 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:44.859280 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:44.860544 5714 hybrid_clock.cc:648] HybridClock initialized: now 1786315064860467 us; error 41 us; skew 500 ppm
I20260809 22:37:44.861001 5714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:44 dist-test-slave-rgb1 krb5kdc[5291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315064, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:44.865376 5714 init.cc:377] Logged in from keytab as oryx/127.28.243.67@KRBTEST.COM (short username oryx)
I20260809 22:37:44.867331 5714 webserver.cc:460] Webserver started at http://127.28.243.67:42971/ using document root <none> and password file <none>
I20260809 22:37:44.868173 5714 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:44.868276 5714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:44.868708 5714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:44.871872 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "a186721b4b354e6494a9d949a782873b"
format_stamp: "Formatted at 2026-08-09 22:37:44 on dist-test-slave-rgb1"
server_key: "846169807621004c36f9f58304549199"
server_key_iv: "c6644d3e4b789043a0d640a0b0ba6ef6"
server_key_version: "encryptionkey@0"
I20260809 22:37:44.872884 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "a186721b4b354e6494a9d949a782873b"
format_stamp: "Formatted at 2026-08-09 22:37:44 on dist-test-slave-rgb1"
server_key: "846169807621004c36f9f58304549199"
server_key_iv: "c6644d3e4b789043a0d640a0b0ba6ef6"
server_key_version: "encryptionkey@0"
I20260809 22:37:44.879217 5714 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:37:44.883555 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:44.885453 5714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 22:37:44.885682 5714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "a186721b4b354e6494a9d949a782873b"
format_stamp: "Formatted at 2026-08-09 22:37:44 on dist-test-slave-rgb1"
server_key: "846169807621004c36f9f58304549199"
server_key_iv: "c6644d3e4b789043a0d640a0b0ba6ef6"
server_key_version: "encryptionkey@0"
I20260809 22:37:44.885975 5714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:44.908113 5714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:44.913537 5714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:44.913869 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:44.916971 5714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:44.917117 5714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:44.918419 5714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:44.918519 5714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 22:37:44 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315064, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:44.945623 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.917869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:52691 (local address 127.28.243.126:36667)
0809 22:37:44.918441 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:44.918448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:44.920229 (+ 1781us) server_negotiation.cc:407] Connection header received
0809 22:37:44.922628 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:44.922638 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:44.922746 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:44.922894 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:44.925590 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.926540 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.929735 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.930202 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.930353 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.936284 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:44.936290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:44.936293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:44.936332 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:44.940513 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.942018 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.942028 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.942034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.942229 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.942975 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.942981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.942985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.943451 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:44.943658 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:44.944842 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0809 22:37:44.945141 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":270,"threads_started":1}
I20260809 22:37:44.946803 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.918559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36667 (local address 127.28.243.67:52691)
0809 22:37:44.919976 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:44.920049 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:44.922139 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:44.923341 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:44.923353 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:44.924305 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:44.925354 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.925387 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.926728 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.926743 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.929530 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.929552 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.930488 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.930497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.930787 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.931730 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:44.931773 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:44.935931 (+ 4158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:44.940774 (+ 4843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.940790 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.940814 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.941797 (+ 983us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.942383 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.942389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.942394 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.942807 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.943659 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:44.943676 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:44.944320 (+ 644us) client_negotiation.cc:769] Sending connection context
0809 22:37:44.944728 (+ 408us) client_negotiation.cc:240] Negotiation successful
0809 22:37:44.945402 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"thread_start_us":801,"threads_started":1}
I20260809 22:37:44.973939 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.954890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:40637 (local address 127.28.243.126:36667)
0809 22:37:44.955138 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:44.955148 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:44.956036 (+ 888us) server_negotiation.cc:407] Connection header received
0809 22:37:44.956350 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:44.956357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:44.956449 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:44.956624 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:44.958496 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.959828 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.963100 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.963558 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.963855 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.967096 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:44.967102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:44.967106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:44.967146 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:44.970681 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.971537 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.971544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.971548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.971703 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:44.972144 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:44.972150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:44.972153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:44.972668 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:44.972845 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:44.973295 (+ 450us) server_negotiation.cc:299] Negotiation successful
0809 22:37:44.973540 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 22:37:44.974121 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:44.954936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36667 (local address 127.28.243.67:40637)
0809 22:37:44.955882 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:44.955914 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:44.956182 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:44.956646 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:44.956653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:44.957589 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:44.958167 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.958194 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.960062 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.960071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.962749 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:44.962779 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:44.963988 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:44.964000 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:44.964267 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:44.965208 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:44.965242 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:44.966785 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:44.970878 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.970885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.970890 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.971371 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.971836 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:44.971841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:44.971845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:44.972001 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:44.972801 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:44.972806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:44.973017 (+ 211us) client_negotiation.cc:769] Sending connection context
0809 22:37:44.973171 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 22:37:44.973644 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":509,"threads_started":1}
I20260809 22:37:44.983733 5714 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:32909
I20260809 22:37:44.983768 5859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:32909 every 8 connection(s)
I20260809 22:37:44.985105 5714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:44.987471 5860 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36667
I20260809 22:37:44.987702 5860 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:44.988349 5860 heartbeater.cc:511] Master 127.28.243.126:36667 requested a full tablet report, sending...
I20260809 22:37:44.989936 5337 ts_manager.cc:194] Registered new tserver with Master: a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:44.990850 5337 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:52691
I20260809 22:37:44.995501 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 5714
I20260809 22:37:44.995664 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:44.996071 29645 external_mini_cluster.cc:1428] Setting key ae4b43aa5c0b2a661cd3dfa92e7ebbb3
I20260809 22:37:44.999854 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 22:37:45 dist-test-slave-rgb1 krb5kdc[5291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 22:37:45 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:45 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 22:37:45.064424 5737 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:45.052408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.28.243.126:36667)
0809 22:37:45.052684 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.052692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.052940 (+ 248us) server_negotiation.cc:407] Connection header received
0809 22:37:45.053121 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.053125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.053206 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.053296 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:45.054751 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.055720 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.057186 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.057669 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.057792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.064133 (+ 6341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41448: BlockingRecv error: recv got EOF from 127.0.0.1:41448 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 09 22:37:45 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:45.085791 5737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.068801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.28.243.126:36667)
0809 22:37:45.069121 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.069127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.069395 (+ 268us) server_negotiation.cc:407] Connection header received
0809 22:37:45.069510 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.069515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.069627 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.069724 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:45.070963 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.071719 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.073223 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.073793 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.073908 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.078991 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:45.078997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:45.079000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:45.079044 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:45.082785 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:45.083582 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:45.083588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:45.083592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:45.083760 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:45.084147 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:45.084152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:45.084155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:45.084597 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:45.084878 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.085169 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.085424 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 22:37:45.090631 5337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:37:45.098063 5337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:45.125167 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.111714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37089 (local address 127.0.0.1:45886)
0809 22:37:45.112713 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.112761 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.112943 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.117878 (+ 4935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.117883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.117910 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:45.118305 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.118320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.119947 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.119953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.121942 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.121956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.123914 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.123921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.124149 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.124217 (+ 68us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.124455 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.124720 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":41,"thread_start_us":4972,"threads_started":1}
I20260809 22:37:45.125759 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.113111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.28.243.66:43927)
0809 22:37:45.114076 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.114086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.114122 (+ 36us) server_negotiation.cc:407] Connection header received
0809 22:37:45.115703 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.115713 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.116030 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.116250 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:45.117251 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.118809 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.121918 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.123627 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.123863 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.124178 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.124947 (+ 769us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.125156 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":1148,"threads_started":1}
I20260809 22:37:45.126528 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.111889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.28.243.65:37089)
0809 22:37:45.117275 (+ 5386us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.117284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.117340 (+ 56us) server_negotiation.cc:407] Connection header received
0809 22:37:45.117444 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.117453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.117695 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.117888 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:45.118513 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.119804 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.122159 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.123665 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.125731 (+ 2066us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.125973 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.126049 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.126230 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5213,"thread_start_us":5074,"threads_started":1}
I20260809 22:37:45.127377 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.112624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:32909 (local address 127.0.0.1:51922)
0809 22:37:45.117325 (+ 4701us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.117350 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.117461 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.118466 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.118473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.118501 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:45.118979 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.118994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.121316 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.121322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.123671 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.123686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.126452 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.126460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.126670 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.126713 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.126823 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.126975 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4573,"thread_start_us":1228,"threads_started":1}
I20260809 22:37:45.128100 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.111916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:43927 (local address 127.0.0.1:33740)
0809 22:37:45.113289 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.113302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.113454 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.116266 (+ 2812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.116274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.116305 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:45.117019 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.117038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.119007 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.119015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.121717 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.121732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.124498 (+ 2766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.124505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.124699 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.124730 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.126198 (+ 1468us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.126351 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":5,"thread_start_us":385,"threads_started":1}
I20260809 22:37:45.128373 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.114204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51922 (local address 127.28.243.67:32909)
0809 22:37:45.115707 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.115716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.117543 (+ 1827us) server_negotiation.cc:407] Connection header received
0809 22:37:45.117664 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.117672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.117924 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.118128 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:45.119330 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.121152 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.123877 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.125859 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.126014 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.126267 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.127703 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.127984 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1331,"thread_start_us":1093,"threads_started":1}
I20260809 22:37:45.131368 5635 tablet_service.cc:1467] Processing CreateTablet for tablet 421d1847808546cb860961a4c8c00beb (DEFAULT_TABLE table=test-table [id=1a322774985f4fd5a255025c94867878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:45.131193 5484 tablet_service.cc:1467] Processing CreateTablet for tablet 421d1847808546cb860961a4c8c00beb (DEFAULT_TABLE table=test-table [id=1a322774985f4fd5a255025c94867878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:45.131892 5781 tablet_service.cc:1467] Processing CreateTablet for tablet 421d1847808546cb860961a4c8c00beb (DEFAULT_TABLE table=test-table [id=1a322774985f4fd5a255025c94867878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:45.132797 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421d1847808546cb860961a4c8c00beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.132790 5635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421d1847808546cb860961a4c8c00beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.133293 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 421d1847808546cb860961a4c8c00beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.147399 5887 tablet_bootstrap.cc:492] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Bootstrap starting.
I20260809 22:37:45.151816 5887 tablet_bootstrap.cc:654] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.153270 5887 log.cc:824] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:45.154340 5889 tablet_bootstrap.cc:492] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Bootstrap starting.
I20260809 22:37:45.154688 5888 tablet_bootstrap.cc:492] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Bootstrap starting.
I20260809 22:37:45.157706 5887 tablet_bootstrap.cc:492] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: No bootstrap required, opened a new log
I20260809 22:37:45.158142 5887 ts_tablet_manager.cc:1397] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260809 22:37:45.158680 5888 tablet_bootstrap.cc:654] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.158684 5889 tablet_bootstrap.cc:654] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.160050 5888 log.cc:824] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:45.160051 5889 log.cc:824] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:45.163239 5889 tablet_bootstrap.cc:492] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: No bootstrap required, opened a new log
I20260809 22:37:45.163522 5889 ts_tablet_manager.cc:1397] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260809 22:37:45.163542 5888 tablet_bootstrap.cc:492] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: No bootstrap required, opened a new log
I20260809 22:37:45.163827 5888 ts_tablet_manager.cc:1397] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260809 22:37:45.170305 5887 raft_consensus.cc:348] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.170851 5887 raft_consensus.cc:374] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.170957 5887 raft_consensus.cc:729] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838c84591d7d42e7b18595471c954de8, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.171661 5889 raft_consensus.cc:348] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.172034 5889 raft_consensus.cc:374] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.172159 5889 raft_consensus.cc:729] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a573f7742744958ab25e4654588469, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.171890 5887 consensus_queue.cc:260] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.172817 5889 consensus_queue.cc:260] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.173452 5709 heartbeater.cc:503] Master 127.28.243.126:36667 was elected leader, sending a full tablet report...
I20260809 22:37:45.174053 5887 ts_tablet_manager.cc:1428] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260809 22:37:45.176664 5889 ts_tablet_manager.cc:1428] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260809 22:37:45.176942 5888 raft_consensus.cc:348] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.177434 5888 raft_consensus.cc:374] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.177552 5888 raft_consensus.cc:729] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a186721b4b354e6494a9d949a782873b, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.178220 5888 consensus_queue.cc:260] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.179427 5860 heartbeater.cc:503] Master 127.28.243.126:36667 was elected leader, sending a full tablet report...
I20260809 22:37:45.180123 5888 ts_tablet_manager.cc:1428] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
W20260809 22:37:45.236924 5861 tablet.cc:2369] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:37:45.291293 5560 tablet.cc:2369] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:45.371958 5894 raft_consensus.cc:482] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:45.372324 5894 raft_consensus.cc:504] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.374274 5894 leader_election.cc:288] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
W20260809 22:37:45.380828 5710 tablet.cc:2369] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:37:45.383102 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.374754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:36371 (local address 127.28.243.67:32909)
0809 22:37:45.375028 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.375036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.375979 (+ 943us) server_negotiation.cc:407] Connection header received
0809 22:37:45.376210 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.376218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.376305 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.376393 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:45.377215 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.377943 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.380696 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.381986 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.382181 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.382259 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.382655 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.382817 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 22:37:45.383447 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.374546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:43927 (local address 127.28.243.65:40819)
0809 22:37:45.375654 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.375679 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.375864 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.376328 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.376333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.376358 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:45.376856 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.376877 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.378395 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.378400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.380533 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.380547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.382473 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.382480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.382681 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.382719 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.382838 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.383076 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":863,"threads_started":1}
I20260809 22:37:45.383494 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.374673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:40819 (local address 127.28.243.66:43927)
0809 22:37:45.374898 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.374904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.375802 (+ 898us) server_negotiation.cc:407] Connection header received
0809 22:37:45.376090 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.376097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.376188 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.376311 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:45.377038 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.378246 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.380746 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.382038 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.382195 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.382259 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.382864 (+ 605us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.383095 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:37:45.384240 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.374587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:32909 (local address 127.28.243.65:36371)
0809 22:37:45.375871 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.375885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.376042 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.376512 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.376516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.376621 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:45.377055 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.377071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.378086 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.378094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.380416 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.380434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.382218 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.382224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.382427 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.382465 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.383457 (+ 992us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.383835 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"mutex_wait_us":2,"spinlock_wait_cycles":30592,"thread_start_us":803,"threads_started":1}
I20260809 22:37:45.384644 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421d1847808546cb860961a4c8c00beb" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8" is_pre_election: true
I20260809 22:37:45.385265 5643 raft_consensus.cc:2454] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:45.385321 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421d1847808546cb860961a4c8c00beb" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b" is_pre_election: true
I20260809 22:37:45.385815 5808 raft_consensus.cc:2454] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:45.386019 5433 leader_election.cc:302] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8; no voters:
I20260809 22:37:45.386463 5894 raft_consensus.cc:2781] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:45.386626 5894 raft_consensus.cc:482] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:45.386721 5894 raft_consensus.cc:3037] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.390215 5894 raft_consensus.cc:504] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.390969 5894 leader_election.cc:288] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Requested vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:45.391618 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421d1847808546cb860961a4c8c00beb" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8"
I20260809 22:37:45.391752 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "421d1847808546cb860961a4c8c00beb" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b"
I20260809 22:37:45.391875 5643 raft_consensus.cc:3037] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.392014 5808 raft_consensus.cc:3037] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.394361 5808 raft_consensus.cc:2454] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:45.394367 5643 raft_consensus.cc:2454] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:45.395057 5433 leader_election.cc:302] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8, a186721b4b354e6494a9d949a782873b; no voters:
I20260809 22:37:45.395735 5894 raft_consensus.cc:2781] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:45.396955 5894 raft_consensus.cc:686] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Becoming Leader. State: Replica: 15a573f7742744958ab25e4654588469, State: Running, Role: LEADER
I20260809 22:37:45.397712 5894 consensus_queue.cc:237] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.409349 5336 catalog_manager.cc:5495] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a573f7742744958ab25e4654588469 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "15a573f7742744958ab25e4654588469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260809 22:37:45.429417 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.423508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.28.243.65:37089)
0809 22:37:45.423757 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.423764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.423889 (+ 125us) server_negotiation.cc:407] Connection header received
0809 22:37:45.424197 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.424202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.424309 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.424483 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:45.424907 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.425640 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.427041 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.427463 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.427608 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.428097 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:45.428756 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:45.428915 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.429030 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.429123 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 22:37:45.438546 5643 raft_consensus.cc:1261] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:45.438621 5808 raft_consensus.cc:1261] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:45.439869 5898 consensus_queue.cc:1035] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:45.440263 5894 consensus_queue.cc:1035] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:45.453823 5902 mvcc.cc:205] Tried to move back new op lower bound from 7316746508024737792 to 7316746507874979840. Current Snapshot: MvccSnapshot[applied={T|T < 7316746508024737792}]
I20260809 22:37:45.466456 5337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:45.466846 5337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:45.471235 5337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Sending DeleteTablet for 3 replicas of tablet 421d1847808546cb860961a4c8c00beb
I20260809 22:37:45.472683 5484 tablet_service.cc:1514] Processing DeleteTablet for tablet 421d1847808546cb860961a4c8c00beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:45 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:45886
I20260809 22:37:45.472685 5635 tablet_service.cc:1514] Processing DeleteTablet for tablet 421d1847808546cb860961a4c8c00beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:45 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:33740
I20260809 22:37:45.473366 5781 tablet_service.cc:1514] Processing DeleteTablet for tablet 421d1847808546cb860961a4c8c00beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:45 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:51922
I20260809 22:37:45.475497 5337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:37:45.476779 5337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:45.479221 5913 tablet_replica.cc:330] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: stopping tablet replica
I20260809 22:37:45.479827 5913 raft_consensus.cc:2229] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:45.480707 5913 raft_consensus.cc:2258] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:45.481608 5912 tablet_replica.cc:330] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: stopping tablet replica
I20260809 22:37:45.482080 5912 raft_consensus.cc:2229] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:45.482599 5912 raft_consensus.cc:2258] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:45.484774 5913 ts_tablet_manager.cc:1905] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:45.487944 5484 tablet_service.cc:1467] Processing CreateTablet for tablet 4e61daf3156145f88f8bfe1151cd6451 (DEFAULT_TABLE table=test-table [id=497f1092f95e4741a59d36d57807b4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:45.488001 5780 tablet_service.cc:1467] Processing CreateTablet for tablet 4e61daf3156145f88f8bfe1151cd6451 (DEFAULT_TABLE table=test-table [id=497f1092f95e4741a59d36d57807b4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:45.488484 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e61daf3156145f88f8bfe1151cd6451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.488691 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e61daf3156145f88f8bfe1151cd6451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.491147 5635 tablet_service.cc:1467] Processing CreateTablet for tablet 4e61daf3156145f88f8bfe1151cd6451 (DEFAULT_TABLE table=test-table [id=497f1092f95e4741a59d36d57807b4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:45.492239 5912 ts_tablet_manager.cc:1905] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:45.496932 5635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e61daf3156145f88f8bfe1151cd6451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.497280 5888 tablet_bootstrap.cc:492] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: Bootstrap starting.
I20260809 22:37:45.499083 5888 tablet_bootstrap.cc:654] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.500234 5889 tablet_bootstrap.cc:492] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: Bootstrap starting.
I20260809 22:37:45.502013 5889 tablet_bootstrap.cc:654] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.505178 5887 tablet_bootstrap.cc:492] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: Bootstrap starting.
I20260809 22:37:45.506853 5887 tablet_bootstrap.cc:654] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.509073 5887 tablet_bootstrap.cc:492] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: No bootstrap required, opened a new log
I20260809 22:37:45.509253 5887 ts_tablet_manager.cc:1397] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260809 22:37:45.509935 5887 raft_consensus.cc:348] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.510140 5887 raft_consensus.cc:374] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.510231 5887 raft_consensus.cc:729] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838c84591d7d42e7b18595471c954de8, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.510555 5887 consensus_queue.cc:260] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.511646 5887 ts_tablet_manager.cc:1428] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 22:37:45.514710 5914 tablet_replica.cc:330] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: stopping tablet replica
I20260809 22:37:45.515242 5914 raft_consensus.cc:2229] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:45.515763 5914 raft_consensus.cc:2258] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:45.516265 5912 ts_tablet_manager.cc:1918] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:45.516455 5912 log.cc:1196] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/421d1847808546cb860961a4c8c00beb
I20260809 22:37:45.517071 5912 ts_tablet_manager.cc:1939] T 421d1847808546cb860961a4c8c00beb P 838c84591d7d42e7b18595471c954de8: Deleting consensus metadata
I20260809 22:37:45.519116 5322 catalog_manager.cc:4841] TS 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927): tablet 421d1847808546cb860961a4c8c00beb (table test-table [id=1a322774985f4fd5a255025c94867878]) successfully deleted
I20260809 22:37:45.520990 5914 ts_tablet_manager.cc:1905] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:45.526403 5889 tablet_bootstrap.cc:492] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: No bootstrap required, opened a new log
I20260809 22:37:45.527642 5889 ts_tablet_manager.cc:1397] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.006s
I20260809 22:37:45.529155 5913 ts_tablet_manager.cc:1918] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:45.529371 5913 log.cc:1196] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/421d1847808546cb860961a4c8c00beb
I20260809 22:37:45.529923 5914 ts_tablet_manager.cc:1918] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:45.530000 5913 ts_tablet_manager.cc:1939] T 421d1847808546cb860961a4c8c00beb P 15a573f7742744958ab25e4654588469: Deleting consensus metadata
I20260809 22:37:45.530093 5914 log.cc:1196] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/421d1847808546cb860961a4c8c00beb
I20260809 22:37:45.530593 5914 ts_tablet_manager.cc:1939] T 421d1847808546cb860961a4c8c00beb P a186721b4b354e6494a9d949a782873b: Deleting consensus metadata
I20260809 22:37:45.528877 5889 raft_consensus.cc:348] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.531389 5889 raft_consensus.cc:374] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.531558 5889 raft_consensus.cc:729] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a573f7742744958ab25e4654588469, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.531988 5324 catalog_manager.cc:4841] TS 15a573f7742744958ab25e4654588469 (127.28.243.65:37089): tablet 421d1847808546cb860961a4c8c00beb (table test-table [id=1a322774985f4fd5a255025c94867878]) successfully deleted
I20260809 22:37:45.531919 5889 consensus_queue.cc:260] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.532385 5321 catalog_manager.cc:4841] TS a186721b4b354e6494a9d949a782873b (127.28.243.67:32909): tablet 421d1847808546cb860961a4c8c00beb (table test-table [id=1a322774985f4fd5a255025c94867878]) successfully deleted
I20260809 22:37:45.532748 5888 tablet_bootstrap.cc:492] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: No bootstrap required, opened a new log
I20260809 22:37:45.532873 5888 ts_tablet_manager.cc:1397] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.006s
I20260809 22:37:45.533461 5889 ts_tablet_manager.cc:1428] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260809 22:37:45.533834 5888 raft_consensus.cc:348] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.534078 5888 raft_consensus.cc:374] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.534197 5888 raft_consensus.cc:729] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a186721b4b354e6494a9d949a782873b, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.534498 5888 consensus_queue.cc:260] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.535241 5888 ts_tablet_manager.cc:1428] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 22:37:45.643741 5906 raft_consensus.cc:482] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:45.644055 5906 raft_consensus.cc:504] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.645085 5906 leader_election.cc:288] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:45.645805 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e61daf3156145f88f8bfe1151cd6451" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b" is_pre_election: true
I20260809 22:37:45.645838 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e61daf3156145f88f8bfe1151cd6451" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8" is_pre_election: true
I20260809 22:37:45.646075 5808 raft_consensus.cc:2454] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:45.646148 5643 raft_consensus.cc:2454] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:45.646625 5431 leader_election.cc:302] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, a186721b4b354e6494a9d949a782873b; no voters:
I20260809 22:37:45.646991 5906 raft_consensus.cc:2781] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:45.647132 5906 raft_consensus.cc:482] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:45.647233 5906 raft_consensus.cc:3037] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.649911 5906 raft_consensus.cc:504] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.650609 5906 leader_election.cc:288] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Requested vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:45.651216 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e61daf3156145f88f8bfe1151cd6451" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b"
I20260809 22:37:45.651199 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e61daf3156145f88f8bfe1151cd6451" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8"
I20260809 22:37:45.651461 5808 raft_consensus.cc:3037] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.651551 5643 raft_consensus.cc:3037] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.653878 5643 raft_consensus.cc:2454] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:45.654376 5433 leader_election.cc:302] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8; no voters:
I20260809 22:37:45.654347 5808 raft_consensus.cc:2454] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:45.654757 5906 raft_consensus.cc:2781] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:45.654925 5906 raft_consensus.cc:686] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Becoming Leader. State: Replica: 15a573f7742744958ab25e4654588469, State: Running, Role: LEADER
I20260809 22:37:45.655210 5906 consensus_queue.cc:237] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.658120 5337 catalog_manager.cc:5495] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a573f7742744958ab25e4654588469 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "15a573f7742744958ab25e4654588469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } health_report { overall_health: UNKNOWN } } }
Aug 09 22:37:45 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315063, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.126@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:45.740880 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.723531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36667 (local address 127.0.0.1:41470)
0809 22:37:45.724615 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.724640 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.724824 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.725343 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.725347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.726069 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:45.726281 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.726291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.727969 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.727975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.729205 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.729241 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.730281 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.730286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.730431 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.731283 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:45.731301 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:45.733714 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:45.737905 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:45.737917 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:45.737923 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:45.738364 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:45.738834 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:45.738838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:45.738840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:45.738986 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:45.739766 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:45.739773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:45.740076 (+ 303us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.740248 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.740461 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":589,"threads_started":1}
I20260809 22:37:45.741443 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.723630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.28.243.126:36667)
0809 22:37:45.724985 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.724992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.725020 (+ 28us) server_negotiation.cc:407] Connection header received
0809 22:37:45.725083 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.725088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.725163 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.725293 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:45.726456 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.727796 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.729392 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.730063 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.730210 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.733959 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:45.733967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:45.733972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:45.734011 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:45.737746 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:45.738504 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:45.738510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:45.738513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:45.738681 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:45.739187 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:45.739192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:45.739195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:45.739625 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:45.739779 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.740403 (+ 624us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.740678 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":1299,"threads_started":1}
I20260809 22:37:45.747156 5337 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:41470:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 22:37:45.748456 5337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:45.756737 5484 tablet_service.cc:1467] Processing CreateTablet for tablet c35216edd8ab4b3eb214d9b5589803fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63f2359a1c214cbbb7f1547d50f93b7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:37:45.757304 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35216edd8ab4b3eb214d9b5589803fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.761394 5780 tablet_service.cc:1467] Processing CreateTablet for tablet c35216edd8ab4b3eb214d9b5589803fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63f2359a1c214cbbb7f1547d50f93b7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:37:45.762027 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35216edd8ab4b3eb214d9b5589803fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.763232 5889 tablet_bootstrap.cc:492] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: Bootstrap starting.
I20260809 22:37:45.763952 5635 tablet_service.cc:1467] Processing CreateTablet for tablet c35216edd8ab4b3eb214d9b5589803fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63f2359a1c214cbbb7f1547d50f93b7d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 22:37:45.764657 5635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35216edd8ab4b3eb214d9b5589803fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:45.765276 5889 tablet_bootstrap.cc:654] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.770267 5888 tablet_bootstrap.cc:492] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: Bootstrap starting.
I20260809 22:37:45.772148 5888 tablet_bootstrap.cc:654] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.774817 5889 tablet_bootstrap.cc:492] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: No bootstrap required, opened a new log
I20260809 22:37:45.775036 5889 ts_tablet_manager.cc:1397] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260809 22:37:45.775898 5887 tablet_bootstrap.cc:492] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: Bootstrap starting.
I20260809 22:37:45.775846 5889 raft_consensus.cc:348] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.776113 5889 raft_consensus.cc:374] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.776199 5889 raft_consensus.cc:729] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a573f7742744958ab25e4654588469, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.776539 5889 consensus_queue.cc:260] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.777040 5906 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.777361 5906 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:37:45.777942 5887 tablet_bootstrap.cc:654] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:45.780790 5889 ts_tablet_manager.cc:1428] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260809 22:37:45.784386 5888 tablet_bootstrap.cc:492] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: No bootstrap required, opened a new log
I20260809 22:37:45.784667 5888 ts_tablet_manager.cc:1397] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260809 22:37:45.785481 5888 raft_consensus.cc:348] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.785760 5888 raft_consensus.cc:374] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.785970 5888 raft_consensus.cc:729] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a186721b4b354e6494a9d949a782873b, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.786260 5888 consensus_queue.cc:260] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.786720 5895 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.787048 5895 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:37:45.788683 5887 tablet_bootstrap.cc:492] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: No bootstrap required, opened a new log
I20260809 22:37:45.788918 5887 ts_tablet_manager.cc:1397] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260809 22:37:45.789877 5887 raft_consensus.cc:348] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.790198 5887 raft_consensus.cc:374] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:45.790268 5887 raft_consensus.cc:729] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838c84591d7d42e7b18595471c954de8, State: Initialized, Role: FOLLOWER
I20260809 22:37:45.790548 5887 consensus_queue.cc:260] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.790732 5888 ts_tablet_manager.cc:1428] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260809 22:37:45.791016 5893 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.791314 5893 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:37:45.792906 5887 ts_tablet_manager.cc:1428] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 22:37:45.868404 5893 raft_consensus.cc:482] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:45.868706 5893 raft_consensus.cc:504] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.870810 5893 leader_election.cc:288] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15a573f7742744958ab25e4654588469 (127.28.243.65:37089), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:45.880439 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.871835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:50183 (local address 127.28.243.65:37089)
0809 22:37:45.872030 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.872037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.873289 (+ 1252us) server_negotiation.cc:407] Connection header received
0809 22:37:45.873560 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.873569 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.873651 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.873704 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:45.874616 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.875745 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.878320 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.879240 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.879460 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.879553 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.879905 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.880054 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 22:37:45.880439 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.871347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37089 (local address 127.28.243.66:50183)
0809 22:37:45.873152 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.873178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.873327 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.873868 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.873872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.873891 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:45.874370 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.874384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.875911 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.875926 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.878124 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.878135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.879502 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.879509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.879712 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.879741 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.879872 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.880038 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":1446,"threads_started":1}
I20260809 22:37:45.880620 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.871440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:40363 (local address 127.28.243.67:32909)
0809 22:37:45.871666 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.871673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.872770 (+ 1097us) server_negotiation.cc:407] Connection header received
0809 22:37:45.873124 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.873132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.873239 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.873344 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:37:45.874006 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.875087 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.877922 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.879259 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.879423 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.879508 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.880148 (+ 640us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.880300 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 22:37:45.881650 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.871313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:32909 (local address 127.28.243.66:40363)
0809 22:37:45.872652 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.872677 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.872906 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.873390 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.873395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.873424 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:37:45.873805 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.873818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.875246 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.875252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.877695 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.877714 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.879721 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.879739 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.879936 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.879972 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.881116 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.881266 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":990,"threads_started":1}
I20260809 22:37:45.881485 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35216edd8ab4b3eb214d9b5589803fa" candidate_uuid: "838c84591d7d42e7b18595471c954de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a573f7742744958ab25e4654588469" is_pre_election: true
I20260809 22:37:45.882082 5509 raft_consensus.cc:2454] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 838c84591d7d42e7b18595471c954de8 in term 0.
I20260809 22:37:45.882665 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35216edd8ab4b3eb214d9b5589803fa" candidate_uuid: "838c84591d7d42e7b18595471c954de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b" is_pre_election: true
I20260809 22:37:45.882884 5584 leader_election.cc:302] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8; no voters:
I20260809 22:37:45.882972 5808 raft_consensus.cc:2454] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 838c84591d7d42e7b18595471c954de8 in term 0.
I20260809 22:37:45.883358 5893 raft_consensus.cc:2781] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:45.883504 5893 raft_consensus.cc:482] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:45.883654 5893 raft_consensus.cc:3037] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.887040 5893 raft_consensus.cc:504] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.887902 5893 leader_election.cc:288] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [CANDIDATE]: Term 1 election: Requested vote from peers 15a573f7742744958ab25e4654588469 (127.28.243.65:37089), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:45.889232 5808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35216edd8ab4b3eb214d9b5589803fa" candidate_uuid: "838c84591d7d42e7b18595471c954de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b"
I20260809 22:37:45.889542 5808 raft_consensus.cc:3037] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.889573 5509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35216edd8ab4b3eb214d9b5589803fa" candidate_uuid: "838c84591d7d42e7b18595471c954de8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15a573f7742744958ab25e4654588469"
I20260809 22:37:45.889866 5509 raft_consensus.cc:3037] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:45.892513 5509 raft_consensus.cc:2454] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 838c84591d7d42e7b18595471c954de8 in term 1.
I20260809 22:37:45.892778 5808 raft_consensus.cc:2454] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 838c84591d7d42e7b18595471c954de8 in term 1.
I20260809 22:37:45.893141 5584 leader_election.cc:302] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8; no voters:
I20260809 22:37:45.893502 5893 raft_consensus.cc:2781] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:45.894066 5893 raft_consensus.cc:686] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [term 1 LEADER]: Becoming Leader. State: Replica: 838c84591d7d42e7b18595471c954de8, State: Running, Role: LEADER
I20260809 22:37:45.894475 5893 consensus_queue.cc:237] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:45.896617 5932 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: TxnStatusTablet state changed. Reason: New leader 838c84591d7d42e7b18595471c954de8. Latest consensus state: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.896955 5932 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:37:45.897785 5935 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 22:37:45.898020 5935 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 22:37:45.899258 5336 catalog_manager.cc:5495] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 838c84591d7d42e7b18595471c954de8 (127.28.243.66). New cstate: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260809 22:37:45.936610 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.927080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:43927 (local address 127.0.0.1:33744)
0809 22:37:45.927312 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:45.927333 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:45.927477 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:45.928849 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:45.928864 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:45.928881 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:37:45.929152 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.929167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.930847 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.930862 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.932804 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:45.932824 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.934184 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.934191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:45.934436 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.934533 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:37:45.935880 (+ 1347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:37:45.935923 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 22:37:45.936035 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 22:37:45.936221 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260809 22:37:45.936728 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.927210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.28.243.66:43927)
0809 22:37:45.928333 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.928340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:45.928365 (+ 25us) server_negotiation.cc:407] Connection header received
0809 22:37:45.928460 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:45.928466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:45.928619 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:45.928779 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:45.929344 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.930667 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.933127 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:45.933948 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:45.934120 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:45.934812 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:45.935737 (+ 925us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:45.935906 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:45.936079 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 22:37:45.936185 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":775,"threads_started":1}
I20260809 22:37:45.942195 5808 raft_consensus.cc:1261] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Refusing update from remote peer 838c84591d7d42e7b18595471c954de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:45.942380 5509 raft_consensus.cc:1261] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Refusing update from remote peer 838c84591d7d42e7b18595471c954de8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:45.943101 5932 consensus_queue.cc:1035] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:45.943486 5893 consensus_queue.cc:1035] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:45.956494 5906 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: TxnStatusTablet state changed. Reason: New leader 838c84591d7d42e7b18595471c954de8. Latest consensus state: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.956897 5906 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:37:45.957289 5895 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: TxnStatusTablet state changed. Reason: New leader 838c84591d7d42e7b18595471c954de8. Latest consensus state: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.957640 5895 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:37:45.959986 5932 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.960281 5932 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:37:45.960873 5932 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.961216 5932 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P 838c84591d7d42e7b18595471c954de8: This TxnStatusTablet replica's current role is: LEADER
I20260809 22:37:45.962693 5901 mvcc.cc:205] Tried to move back new op lower bound from 7316746510088310784 to 7316746509907906560. Current Snapshot: MvccSnapshot[applied={T|T < 7316746510088310784}]
I20260809 22:37:45.963156 5903 mvcc.cc:205] Tried to move back new op lower bound from 7316746510088310784 to 7316746509907906560. Current Snapshot: MvccSnapshot[applied={T|T < 7316746510088310784}]
I20260809 22:37:45.977299 5895 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.977625 5895 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P a186721b4b354e6494a9d949a782873b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:37:45.987443 5906 tablet_replica.cc:439] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "838c84591d7d42e7b18595471c954de8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } } }
I20260809 22:37:45.987828 5906 tablet_replica.cc:442] T c35216edd8ab4b3eb214d9b5589803fa P 15a573f7742744958ab25e4654588469: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 22:37:46.013643 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.993917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:34737 (local address 127.28.243.66:43927)
0809 22:37:45.994119 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:45.994125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:46.006147 (+ 12022us) server_negotiation.cc:407] Connection header received
0809 22:37:46.006365 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:46.006372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:46.006491 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:46.006673 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:46.007253 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.008655 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.011101 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.011909 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.012082 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.012727 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:46.012894 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:46.013000 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:46.013190 (+ 190us) server_negotiation.cc:299] Negotiation successful
0809 22:37:46.013289 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 22:37:46.013787 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:45.993474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:43927 (local address 127.28.243.65:34737)
0809 22:37:46.005951 (+ 12477us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:46.005984 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:46.006204 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:46.006691 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:46.006696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:46.006712 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:37:46.007049 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:46.007061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.008922 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.008929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:46.010883 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:46.010906 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.012167 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.012174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:46.012391 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.012494 (+ 103us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:37:46.013029 (+ 535us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:37:46.013043 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 22:37:46.013148 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 22:37:46.013333 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12212,"thread_start_us":12002,"threads_started":1}
I20260809 22:37:46.027510 5808 raft_consensus.cc:1261] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:46.027551 5643 raft_consensus.cc:1261] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:46.028162 5898 consensus_queue.cc:1035] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:46.028501 5906 consensus_queue.cc:1035] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:46.078444 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:46.070932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:60499 (local address 127.28.243.65:37089)
0809 22:37:46.071130 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:46.071137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:46.071414 (+ 277us) server_negotiation.cc:407] Connection header received
0809 22:37:46.071652 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:46.071659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:46.071745 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:46.071840 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:46.072779 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.073881 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.076532 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.077005 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.077155 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.077631 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:46.077738 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:46.077838 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:46.078033 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 22:37:46.078119 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 22:37:46.078649 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:46.070566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37089 (local address 127.28.243.66:60499)
0809 22:37:46.071313 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:46.071337 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:46.071494 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:46.072030 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:46.072048 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:46.072061 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 22:37:46.072462 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:46.072474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.074044 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.074053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:46.076352 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:46.076370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.077220 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.077227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:46.077367 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.077430 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 22:37:46.077840 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 22:37:46.077854 (+ 14us) client_negotiation.cc:769] Sending connection context
0809 22:37:46.078012 (+ 158us) client_negotiation.cc:240] Negotiation successful
0809 22:37:46.078169 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":577,"threads_started":1}
I20260809 22:37:46.126830 5336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:46.127116 5336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:46.131810 5336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Sending DeleteTablet for 3 replicas of tablet 4e61daf3156145f88f8bfe1151cd6451
I20260809 22:37:46.132911 5635 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61daf3156145f88f8bfe1151cd6451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:33740
I20260809 22:37:46.133034 5780 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61daf3156145f88f8bfe1151cd6451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:51922
I20260809 22:37:46.133731 5954 tablet_replica.cc:330] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: stopping tablet replica
I20260809 22:37:46.133795 5484 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e61daf3156145f88f8bfe1151cd6451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:45886
I20260809 22:37:46.133968 5954 raft_consensus.cc:2229] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:46.134349 5954 raft_consensus.cc:2258] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.140921 5954 ts_tablet_manager.cc:1905] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.150607 5954 ts_tablet_manager.cc:1918] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:37:46.150874 5954 log.cc:1196] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/4e61daf3156145f88f8bfe1151cd6451
I20260809 22:37:46.151526 5954 ts_tablet_manager.cc:1939] T 4e61daf3156145f88f8bfe1151cd6451 P 838c84591d7d42e7b18595471c954de8: Deleting consensus metadata
I20260809 22:37:46.153005 5322 catalog_manager.cc:4841] TS 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927): tablet 4e61daf3156145f88f8bfe1151cd6451 (table test-table [id=497f1092f95e4741a59d36d57807b4ba]) successfully deleted
I20260809 22:37:46.154577 5957 tablet_replica.cc:330] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: stopping tablet replica
I20260809 22:37:46.154917 5957 raft_consensus.cc:2229] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:46.155447 5957 raft_consensus.cc:2258] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.156826 5957 ts_tablet_manager.cc:1905] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.157395 5956 tablet_replica.cc:330] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: stopping tablet replica
I20260809 22:37:46.157660 5956 raft_consensus.cc:2229] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:46.157984 5956 raft_consensus.cc:2258] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.161516 5956 ts_tablet_manager.cc:1905] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.167297 5957 ts_tablet_manager.cc:1918] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:37:46.167479 5957 log.cc:1196] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/4e61daf3156145f88f8bfe1151cd6451
I20260809 22:37:46.168071 5957 ts_tablet_manager.cc:1939] T 4e61daf3156145f88f8bfe1151cd6451 P 15a573f7742744958ab25e4654588469: Deleting consensus metadata
I20260809 22:37:46.169402 5956 ts_tablet_manager.cc:1918] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:37:46.169441 5324 catalog_manager.cc:4841] TS 15a573f7742744958ab25e4654588469 (127.28.243.65:37089): tablet 4e61daf3156145f88f8bfe1151cd6451 (table test-table [id=497f1092f95e4741a59d36d57807b4ba]) successfully deleted
I20260809 22:37:46.169672 5956 log.cc:1196] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/4e61daf3156145f88f8bfe1151cd6451
I20260809 22:37:46.170225 5956 ts_tablet_manager.cc:1939] T 4e61daf3156145f88f8bfe1151cd6451 P a186721b4b354e6494a9d949a782873b: Deleting consensus metadata
I20260809 22:37:46.171370 5321 catalog_manager.cc:4841] TS a186721b4b354e6494a9d949a782873b (127.28.243.67:32909): tablet 4e61daf3156145f88f8bfe1151cd6451 (table test-table [id=497f1092f95e4741a59d36d57807b4ba]) successfully deleted
Aug 09 22:37:46 dist-test-slave-rgb1 krb5kdc[5291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 22:37:46 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:46 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 22:37:46.216739 5925 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:46.203743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.28.243.126:36667)
0809 22:37:46.203980 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:46.203988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:46.204514 (+ 526us) server_negotiation.cc:407] Connection header received
0809 22:37:46.204744 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:46.204749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:46.204837 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:46.204928 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:46.206889 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.207767 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.209680 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.210136 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.210255 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.216358 (+ 6103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41480: BlockingRecv error: recv got EOF from 127.0.0.1:41480 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 22:37:46 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM
I20260809 22:37:46.235105 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:46.219782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.28.243.126:36667)
0809 22:37:46.219995 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:46.220003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:46.220360 (+ 357us) server_negotiation.cc:407] Connection header received
0809 22:37:46.220625 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:46.220631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:46.220714 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:46.220835 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:46.222040 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.222918 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.224288 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.224798 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.224942 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.229244 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:46.229250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:46.229253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:46.229292 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:46.232182 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:46.232967 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:46.232973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:46.232979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:46.233124 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:46.233538 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:46.233544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:46.233547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:46.233862 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:46.234156 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:46.234473 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 22:37:46.234744 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:37:46.239285 5336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:37:46.240110 5336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:46.247915 5635 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 (DEFAULT_TABLE table=test-table [id=74cbdd7851a94167969c16e0e4e54878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:46.247910 5484 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 (DEFAULT_TABLE table=test-table [id=74cbdd7851a94167969c16e0e4e54878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:46.248306 5780 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 (DEFAULT_TABLE table=test-table [id=74cbdd7851a94167969c16e0e4e54878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:46.248427 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4dd92f0b8240639a13fb29fa0a02a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:46.248477 5635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4dd92f0b8240639a13fb29fa0a02a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:46.249303 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b4dd92f0b8240639a13fb29fa0a02a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:46.256083 5889 tablet_bootstrap.cc:492] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: Bootstrap starting.
I20260809 22:37:46.256084 5887 tablet_bootstrap.cc:492] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: Bootstrap starting.
I20260809 22:37:46.257155 5888 tablet_bootstrap.cc:492] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: Bootstrap starting.
I20260809 22:37:46.257957 5889 tablet_bootstrap.cc:654] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:46.257956 5887 tablet_bootstrap.cc:654] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:46.258912 5888 tablet_bootstrap.cc:654] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:46.269141 5888 tablet_bootstrap.cc:492] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: No bootstrap required, opened a new log
I20260809 22:37:46.269423 5888 ts_tablet_manager.cc:1397] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260809 22:37:46.271049 5887 tablet_bootstrap.cc:492] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: No bootstrap required, opened a new log
I20260809 22:37:46.271248 5887 ts_tablet_manager.cc:1397] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260809 22:37:46.271405 5889 tablet_bootstrap.cc:492] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: No bootstrap required, opened a new log
I20260809 22:37:46.271579 5889 ts_tablet_manager.cc:1397] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260809 22:37:46.271258 5888 raft_consensus.cc:348] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.271816 5888 raft_consensus.cc:374] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:46.271974 5888 raft_consensus.cc:729] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a186721b4b354e6494a9d949a782873b, State: Initialized, Role: FOLLOWER
I20260809 22:37:46.272157 5887 raft_consensus.cc:348] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.272502 5887 raft_consensus.cc:374] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:46.272360 5889 raft_consensus.cc:348] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.272637 5889 raft_consensus.cc:374] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:46.272653 5887 raft_consensus.cc:729] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838c84591d7d42e7b18595471c954de8, State: Initialized, Role: FOLLOWER
I20260809 22:37:46.272739 5889 raft_consensus.cc:729] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a573f7742744958ab25e4654588469, State: Initialized, Role: FOLLOWER
I20260809 22:37:46.272939 5887 consensus_queue.cc:260] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.273005 5889 consensus_queue.cc:260] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.273561 5889 ts_tablet_manager.cc:1428] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 22:37:46.273670 5887 ts_tablet_manager.cc:1428] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 22:37:46.273005 5888 consensus_queue.cc:260] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.277829 5888 ts_tablet_manager.cc:1428] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260809 22:37:46.549283 5948 raft_consensus.cc:482] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:46.549549 5948 raft_consensus.cc:504] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.550599 5948 leader_election.cc:288] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:46.551301 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4dd92f0b8240639a13fb29fa0a02a3" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8" is_pre_election: true
I20260809 22:37:46.551291 5807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4dd92f0b8240639a13fb29fa0a02a3" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b" is_pre_election: true
I20260809 22:37:46.551574 5807 raft_consensus.cc:2454] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:46.551576 5643 raft_consensus.cc:2454] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:46.552197 5433 leader_election.cc:302] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8, a186721b4b354e6494a9d949a782873b; no voters:
I20260809 22:37:46.552536 5948 raft_consensus.cc:2781] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:46.552794 5948 raft_consensus.cc:482] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:46.552928 5948 raft_consensus.cc:3037] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:46.555516 5948 raft_consensus.cc:504] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.556169 5948 leader_election.cc:288] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Requested vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:46.556851 5807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4dd92f0b8240639a13fb29fa0a02a3" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b"
I20260809 22:37:46.556923 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4dd92f0b8240639a13fb29fa0a02a3" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8"
I20260809 22:37:46.557127 5643 raft_consensus.cc:3037] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:46.557174 5807 raft_consensus.cc:3037] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:46.559465 5807 raft_consensus.cc:2454] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:46.559495 5643 raft_consensus.cc:2454] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:46.560066 5431 leader_election.cc:302] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8, a186721b4b354e6494a9d949a782873b; no voters:
I20260809 22:37:46.560451 5948 raft_consensus.cc:2781] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:46.560696 5948 raft_consensus.cc:686] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Becoming Leader. State: Replica: 15a573f7742744958ab25e4654588469, State: Running, Role: LEADER
I20260809 22:37:46.561116 5948 consensus_queue.cc:237] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.564286 5335 catalog_manager.cc:5495] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a573f7742744958ab25e4654588469 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "15a573f7742744958ab25e4654588469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260809 22:37:46.586274 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:46.580385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.28.243.65:37089)
0809 22:37:46.581118 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:46.581123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:46.581141 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:46.581205 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:46.581210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:46.581278 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:46.581404 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:37:46.581838 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.582671 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.584159 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.584694 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.584865 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.585390 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:37:46.585521 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:37:46.585634 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:46.585804 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 22:37:46.585912 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":363,"threads_started":1}
I20260809 22:37:46.589203 5643 raft_consensus.cc:1261] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:46.589263 5807 raft_consensus.cc:1261] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:46.590076 5948 consensus_queue.cc:1035] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:46.590438 5898 consensus_queue.cc:1035] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:46.603895 5336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:46.604270 5336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:46.608377 5336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Sending DeleteTablet for 3 replicas of tablet 2b4dd92f0b8240639a13fb29fa0a02a3
I20260809 22:37:46.609476 5635 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:33740
I20260809 22:37:46.609706 5780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:51922
I20260809 22:37:46.609962 5954 tablet_replica.cc:330] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: stopping tablet replica
I20260809 22:37:46.609898 5484 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:45886
I20260809 22:37:46.610077 5956 tablet_replica.cc:330] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: stopping tablet replica
I20260809 22:37:46.610215 5954 raft_consensus.cc:2229] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:46.610260 5956 raft_consensus.cc:2229] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:46.610484 5954 raft_consensus.cc:2258] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.610569 5956 raft_consensus.cc:2258] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.610800 5957 tablet_replica.cc:330] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: stopping tablet replica
I20260809 22:37:46.610831 5780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:51922
I20260809 22:37:46.611038 5957 raft_consensus.cc:2229] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:46.611433 5957 raft_consensus.cc:2258] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.612464 5336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 22:37:46.612833 5956 ts_tablet_manager.cc:1905] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 22:37:46.613314 5336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:37:46.613447 5954 ts_tablet_manager.cc:1905] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.617424 5957 ts_tablet_manager.cc:1905] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.618919 5954 ts_tablet_manager.cc:1918] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:46.619100 5954 log.cc:1196] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/2b4dd92f0b8240639a13fb29fa0a02a3
I20260809 22:37:46.619689 5954 ts_tablet_manager.cc:1939] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 838c84591d7d42e7b18595471c954de8: Deleting consensus metadata
I20260809 22:37:46.620898 5322 catalog_manager.cc:4841] TS 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927): tablet 2b4dd92f0b8240639a13fb29fa0a02a3 (table test-table [id=74cbdd7851a94167969c16e0e4e54878]) successfully deleted
I20260809 22:37:46.622833 5484 tablet_service.cc:1467] Processing CreateTablet for tablet 3838e8450da243d7926b1da0f3bb0b97 (DEFAULT_TABLE table=test-table [id=c9c02fb3506a4ed4b692f7f3e8881115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:46.623435 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3838e8450da243d7926b1da0f3bb0b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:46.625856 5780 tablet_service.cc:1467] Processing CreateTablet for tablet 3838e8450da243d7926b1da0f3bb0b97 (DEFAULT_TABLE table=test-table [id=c9c02fb3506a4ed4b692f7f3e8881115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:46.626314 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3838e8450da243d7926b1da0f3bb0b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:46.627722 5956 ts_tablet_manager.cc:1918] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:46.628842 5956 log.cc:1196] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/2b4dd92f0b8240639a13fb29fa0a02a3
I20260809 22:37:46.629443 5956 ts_tablet_manager.cc:1939] T 2b4dd92f0b8240639a13fb29fa0a02a3 P a186721b4b354e6494a9d949a782873b: Deleting consensus metadata
I20260809 22:37:46.630254 5957 ts_tablet_manager.cc:1918] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:37:46.630432 5957 log.cc:1196] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/2b4dd92f0b8240639a13fb29fa0a02a3
I20260809 22:37:46.630842 5321 catalog_manager.cc:4841] TS a186721b4b354e6494a9d949a782873b (127.28.243.67:32909): tablet 2b4dd92f0b8240639a13fb29fa0a02a3 (table test-table [id=74cbdd7851a94167969c16e0e4e54878]) successfully deleted
I20260809 22:37:46.630932 5957 ts_tablet_manager.cc:1939] T 2b4dd92f0b8240639a13fb29fa0a02a3 P 15a573f7742744958ab25e4654588469: Deleting consensus metadata
I20260809 22:37:46.630971 5635 tablet_service.cc:1467] Processing CreateTablet for tablet 3838e8450da243d7926b1da0f3bb0b97 (DEFAULT_TABLE table=test-table [id=c9c02fb3506a4ed4b692f7f3e8881115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:37:46.631475 5635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3838e8450da243d7926b1da0f3bb0b97. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 22:37:46.631680 5321 catalog_manager.cc:4808] TS a186721b4b354e6494a9d949a782873b (127.28.243.67:32909): delete failed for tablet 2b4dd92f0b8240639a13fb29fa0a02a3 because the tablet was not found. No further retry: Not found: Tablet not found: 2b4dd92f0b8240639a13fb29fa0a02a3
I20260809 22:37:46.632264 5324 catalog_manager.cc:4841] TS 15a573f7742744958ab25e4654588469 (127.28.243.65:37089): tablet 2b4dd92f0b8240639a13fb29fa0a02a3 (table test-table [id=74cbdd7851a94167969c16e0e4e54878]) successfully deleted
I20260809 22:37:46.633369 5888 tablet_bootstrap.cc:492] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: Bootstrap starting.
I20260809 22:37:46.634944 5888 tablet_bootstrap.cc:654] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:46.637221 5888 tablet_bootstrap.cc:492] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: No bootstrap required, opened a new log
I20260809 22:37:46.637393 5887 tablet_bootstrap.cc:492] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: Bootstrap starting.
I20260809 22:37:46.637423 5888 ts_tablet_manager.cc:1397] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:46.637776 5889 tablet_bootstrap.cc:492] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: Bootstrap starting.
I20260809 22:37:46.638186 5888 raft_consensus.cc:348] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.638427 5888 raft_consensus.cc:374] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:46.638510 5888 raft_consensus.cc:729] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a186721b4b354e6494a9d949a782873b, State: Initialized, Role: FOLLOWER
I20260809 22:37:46.638784 5888 consensus_queue.cc:260] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.639312 5887 tablet_bootstrap.cc:654] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:46.639417 5888 ts_tablet_manager.cc:1428] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260809 22:37:46.639770 5889 tablet_bootstrap.cc:654] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:46.641904 5887 tablet_bootstrap.cc:492] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: No bootstrap required, opened a new log
I20260809 22:37:46.642163 5887 ts_tablet_manager.cc:1397] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260809 22:37:46.643249 5887 raft_consensus.cc:348] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.643630 5887 raft_consensus.cc:374] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:46.643748 5887 raft_consensus.cc:729] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838c84591d7d42e7b18595471c954de8, State: Initialized, Role: FOLLOWER
I20260809 22:37:46.644133 5887 consensus_queue.cc:260] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.645002 5887 ts_tablet_manager.cc:1428] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260809 22:37:46.646778 5889 tablet_bootstrap.cc:492] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: No bootstrap required, opened a new log
I20260809 22:37:46.647001 5889 ts_tablet_manager.cc:1397] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260809 22:37:46.647850 5889 raft_consensus.cc:348] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.648104 5889 raft_consensus.cc:374] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:46.648216 5889 raft_consensus.cc:729] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15a573f7742744958ab25e4654588469, State: Initialized, Role: FOLLOWER
I20260809 22:37:46.648521 5889 consensus_queue.cc:260] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.649171 5889 ts_tablet_manager.cc:1428] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260809 22:37:46.651111 5980 raft_consensus.cc:482] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:37:46.651333 5980 raft_consensus.cc:504] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.652516 5980 leader_election.cc:288] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:46.652603 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3838e8450da243d7926b1da0f3bb0b97" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8" is_pre_election: true
I20260809 22:37:46.652674 5807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3838e8450da243d7926b1da0f3bb0b97" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b" is_pre_election: true
I20260809 22:37:46.652935 5643 raft_consensus.cc:2454] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:46.652967 5807 raft_consensus.cc:2454] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 0.
I20260809 22:37:46.653506 5433 leader_election.cc:302] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8; no voters:
I20260809 22:37:46.653831 5980 raft_consensus.cc:2781] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:37:46.653980 5980 raft_consensus.cc:482] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:37:46.654109 5980 raft_consensus.cc:3037] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:46.656448 5980 raft_consensus.cc:504] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.657233 5980 leader_election.cc:288] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Requested vote from peers 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927), a186721b4b354e6494a9d949a782873b (127.28.243.67:32909)
I20260809 22:37:46.657760 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3838e8450da243d7926b1da0f3bb0b97" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838c84591d7d42e7b18595471c954de8"
I20260809 22:37:46.657892 5807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3838e8450da243d7926b1da0f3bb0b97" candidate_uuid: "15a573f7742744958ab25e4654588469" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a186721b4b354e6494a9d949a782873b"
I20260809 22:37:46.658002 5643 raft_consensus.cc:3037] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:46.658125 5807 raft_consensus.cc:3037] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:46.660647 5643 raft_consensus.cc:2454] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:46.660832 5807 raft_consensus.cc:2454] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15a573f7742744958ab25e4654588469 in term 1.
I20260809 22:37:46.661137 5433 leader_election.cc:302] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15a573f7742744958ab25e4654588469, 838c84591d7d42e7b18595471c954de8; no voters:
I20260809 22:37:46.661463 5980 raft_consensus.cc:2781] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:46.661655 5980 raft_consensus.cc:686] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Becoming Leader. State: Replica: 15a573f7742744958ab25e4654588469, State: Running, Role: LEADER
I20260809 22:37:46.661978 5980 consensus_queue.cc:237] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } }
I20260809 22:37:46.665026 5337 catalog_manager.cc:5495] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15a573f7742744958ab25e4654588469 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "15a573f7742744958ab25e4654588469" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15a573f7742744958ab25e4654588469" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260809 22:37:46.695926 5807 raft_consensus.cc:1261] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:46.695925 5643 raft_consensus.cc:1261] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Refusing update from remote peer 15a573f7742744958ab25e4654588469: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:37:46.696601 5980 consensus_queue.cc:1035] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a186721b4b354e6494a9d949a782873b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:46.696928 5898 consensus_queue.cc:1035] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838c84591d7d42e7b18595471c954de8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 43927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:37:46.743846 5336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:46.744149 5336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:37:46.747485 5336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37562621482d4d3097cfa2c40c4cfba9: Sending DeleteTablet for 3 replicas of tablet 3838e8450da243d7926b1da0f3bb0b97
I20260809 22:37:46.748505 5484 tablet_service.cc:1514] Processing DeleteTablet for tablet 3838e8450da243d7926b1da0f3bb0b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:45886
I20260809 22:37:46.748817 5780 tablet_service.cc:1514] Processing DeleteTablet for tablet 3838e8450da243d7926b1da0f3bb0b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:51922
I20260809 22:37:46.748814 5635 tablet_service.cc:1514] Processing DeleteTablet for tablet 3838e8450da243d7926b1da0f3bb0b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:37:46 UTC) from {username='oryx', principal='oryx/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:33740
I20260809 22:37:46.749177 5957 tablet_replica.cc:330] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: stopping tablet replica
I20260809 22:37:46.749178 5956 tablet_replica.cc:330] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: stopping tablet replica
I20260809 22:37:46.749378 5956 raft_consensus.cc:2229] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:46.749388 5957 raft_consensus.cc:2229] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:37:46.749583 5954 tablet_replica.cc:330] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: stopping tablet replica
I20260809 22:37:46.749645 5956 raft_consensus.cc:2258] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.749802 5954 raft_consensus.cc:2229] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:37:46.749788 5957 raft_consensus.cc:2258] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.750043 5954 raft_consensus.cc:2258] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:37:46.750903 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 5414
I20260809 22:37:46.751060 5956 ts_tablet_manager.cc:1905] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.751060 5957 ts_tablet_manager.cc:1905] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.753093 5954 ts_tablet_manager.cc:1905] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:37:46.758100 5956 ts_tablet_manager.cc:1918] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:37:46.758294 5956 log.cc:1196] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/3838e8450da243d7926b1da0f3bb0b97
I20260809 22:37:46.758281 5957 ts_tablet_manager.cc:1918] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:37:46.758410 5957 log.cc:1196] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/3838e8450da243d7926b1da0f3bb0b97
I20260809 22:37:46.758776 5956 ts_tablet_manager.cc:1939] T 3838e8450da243d7926b1da0f3bb0b97 P a186721b4b354e6494a9d949a782873b: Deleting consensus metadata
I20260809 22:37:46.758903 5957 ts_tablet_manager.cc:1939] T 3838e8450da243d7926b1da0f3bb0b97 P 15a573f7742744958ab25e4654588469: Deleting consensus metadata
I20260809 22:37:46.760082 5321 catalog_manager.cc:4841] TS a186721b4b354e6494a9d949a782873b (127.28.243.67:32909): tablet 3838e8450da243d7926b1da0f3bb0b97 (table test-table [id=c9c02fb3506a4ed4b692f7f3e8881115]) successfully deleted
I20260809 22:37:46.760428 5324 catalog_manager.cc:4841] TS 15a573f7742744958ab25e4654588469 (127.28.243.65:37089): tablet 3838e8450da243d7926b1da0f3bb0b97 (table test-table [id=c9c02fb3506a4ed4b692f7f3e8881115]) successfully deleted
I20260809 22:37:46.763031 5954 ts_tablet_manager.cc:1918] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 22:37:46.763164 5954 log.cc:1196] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/3838e8450da243d7926b1da0f3bb0b97
I20260809 22:37:46.763764 5954 ts_tablet_manager.cc:1939] T 3838e8450da243d7926b1da0f3bb0b97 P 838c84591d7d42e7b18595471c954de8: Deleting consensus metadata
I20260809 22:37:46.764958 5322 catalog_manager.cc:4841] TS 838c84591d7d42e7b18595471c954de8 (127.28.243.66:43927): tablet 3838e8450da243d7926b1da0f3bb0b97 (table test-table [id=c9c02fb3506a4ed4b692f7f3e8881115]) successfully deleted
Aug 09 22:37:46 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.65@KRBTEST.COM
I20260809 22:37:46.777415 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:46.759932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.28.243.65:37089)
0809 22:37:46.760109 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:46.760118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:46.761667 (+ 1549us) server_negotiation.cc:407] Connection header received
0809 22:37:46.761812 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:46.761818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:46.761911 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:46.762053 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:46.763447 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.764549 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.766165 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.766672 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.766802 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.771303 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:46.771322 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:46.771339 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:46.771383 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:46.774643 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:46.775314 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:46.775322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:46.775324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:46.775457 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:46.775852 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:46.775856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:46.775859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:46.776301 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:46.776639 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:46.776890 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 22:37:46.777165 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260809 22:37:46.778201 5554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:46.911041 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 5414
I20260809 22:37:46.937784 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 5563
Aug 09 22:37:46 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.66@KRBTEST.COM
I20260809 22:37:46.955291 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:46.938705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.28.243.66:43927)
0809 22:37:46.939348 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:46.939352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:46.939398 (+ 46us) server_negotiation.cc:407] Connection header received
0809 22:37:46.939475 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:46.939479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:46.939542 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:46.939798 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:46.941386 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.942130 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.943398 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:46.943809 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:46.943958 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:46.948817 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:46.948834 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:46.948852 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:46.948895 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:46.952118 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:46.952924 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:46.952932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:46.952935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:46.953117 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:46.953541 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:46.953545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:46.953547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:46.954087 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:46.954318 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:46.954683 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 22:37:46.954909 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":417,"threads_started":1}
I20260809 22:37:46.955994 5703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:47.085449 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 5563
I20260809 22:37:47.111985 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 5714
Aug 09 22:37:47 dist-test-slave-rgb1 krb5kdc[5291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.243.67@KRBTEST.COM
I20260809 22:37:47.130986 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:47.113198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.28.243.67:32909)
0809 22:37:47.113897 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:47.113902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:47.113923 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:37:47.113988 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:47.113992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:47.114067 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:47.114179 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:47.115761 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:47.116844 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:47.118297 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:47.118786 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:47.118904 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:47.124210 (+ 5306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:47.124227 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:47.124244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:47.124284 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:47.127657 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:47.128783 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:47.128792 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:47.128794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:47.128957 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:47.129397 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:47.129402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:47.129404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:47.129805 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:47.130123 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:47.130359 (+ 236us) server_negotiation.cc:299] Negotiation successful
0809 22:37:47.130559 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":348,"threads_started":1}
I20260809 22:37:47.131966 5853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:47.240514 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 5714
I20260809 22:37:47.261896 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 5304
I20260809 22:37:47.263283 5381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:47.386036 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 5304
2026-08-09T22:37:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6282 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:47 dist-test-slave-rgb1 krb5kdc[6000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:47 dist-test-slave-rgb1 krb5kdc[6000](info): set up 2 sockets
Aug 09 22:37:47 dist-test-slave-rgb1 krb5kdc[6000](info): commencing operation
krb5kdc: starting...
W20260809 22:37:48.588207 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.005s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:48 dist-test-slave-rgb1 krb5kdc[6000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:48.890244 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:46781
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:46781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:49.248116 6015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:49.248448 6015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:49.248544 6015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:49.256071 6015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:49.256201 6015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:49.256250 6015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:49.256301 6015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:49.256350 6015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:49.267156 6015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:46781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:49.268518 6015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:49.269897 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:49.283080 6023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:49.283018 6020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:49.283046 6021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:49.284703 6015 server_base.cc:1034] running on GCE node
I20260809 22:37:49.285544 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:49.287019 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:49.288237 6015 hybrid_clock.cc:648] HybridClock initialized: now 1786315069288190 us; error 35 us; skew 500 ppm
I20260809 22:37:49.288712 6015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T22:37:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1973 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:37:49 dist-test-slave-rgb1 krb5kdc[6030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:37:49 dist-test-slave-rgb1 krb5kdc[6030](info): set up 2 sockets
Aug 09 22:37:49 dist-test-slave-rgb1 krb5kdc[6030](info): commencing operation
krb5kdc: starting...
W20260809 22:37:50.603152 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:37:50 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:37:50.906445 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41471
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:51.266630 6045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:51.266942 6045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:51.267084 6045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:51.274564 6045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:51.274694 6045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:51.274778 6045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:51.274873 6045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:51.274961 6045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:51.285331 6045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:51.286767 6045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:51.288110 6045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:51.300369 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:51.300438 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:51.300472 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:51.301643 6045 server_base.cc:1034] running on GCE node
I20260809 22:37:51.302496 6045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:51.304059 6045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:51.305307 6045 hybrid_clock.cc:648] HybridClock initialized: now 1786315071305240 us; error 60 us; skew 500 ppm
I20260809 22:37:51.305712 6045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:51 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315071, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:51.309083 6045 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:51.310817 6045 webserver.cc:460] Webserver started at http://127.28.243.126:33805/ using document root <none> and password file <none>
I20260809 22:37:51.311597 6045 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:51.311988 6045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:51.312382 6045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:51.315585 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "a1cedb8a951644df9ee18e56b11d9cd5"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "373f3f95e4ec14f1b71cc40d2d7d463b"
server_key_iv: "e0c371beb1931781b87a9602f507aac4"
server_key_version: "encryptionkey@0"
I20260809 22:37:51.316401 6045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "a1cedb8a951644df9ee18e56b11d9cd5"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "373f3f95e4ec14f1b71cc40d2d7d463b"
server_key_iv: "e0c371beb1931781b87a9602f507aac4"
server_key_version: "encryptionkey@0"
I20260809 22:37:51.322126 6045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 22:37:51.326052 6061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:51.327611 6045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:37:51.327811 6045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "a1cedb8a951644df9ee18e56b11d9cd5"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "373f3f95e4ec14f1b71cc40d2d7d463b"
server_key_iv: "e0c371beb1931781b87a9602f507aac4"
server_key_version: "encryptionkey@0"
I20260809 22:37:51.328042 6045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:51.343758 6045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:51.348623 6045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:51.348937 6045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:51.369478 6045 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:41471
I20260809 22:37:51.369531 6115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:41471 every 8 connection(s)
I20260809 22:37:51.371173 6045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:51.375350 6116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:37:51.377971 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6045
I20260809 22:37:51.378221 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:37:51.378892 29645 external_mini_cluster.cc:1428] Setting key 1d1515bfcec63edb9d36ee2707576c11
I20260809 22:37:51.389051 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Bootstrap starting.
I20260809 22:37:51.392833 6116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Neither blocks nor log segments found. Creating new log.
I20260809 22:37:51.393888 6116 log.cc:824] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Log is configured to *not* fsync() on all Append() calls
Aug 09 22:37:51 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:51.397435 6116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: No bootstrap required, opened a new log
I20260809 22:37:51.407490 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:51.381136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.28.243.126:41471)
0809 22:37:51.381932 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:51.381947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:51.381995 (+ 48us) server_negotiation.cc:407] Connection header received
0809 22:37:51.384628 (+ 2633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:51.384659 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:51.385292 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:51.385738 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:51.387174 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:51.389233 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:51.390651 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:51.391720 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:51.391930 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:51.397101 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:51.397114 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:51.397134 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:51.397185 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:51.401718 (+ 4533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:51.402930 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:51.402941 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:51.402950 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:51.403245 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:51.403998 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:51.404005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:51.404008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:51.404878 (+ 870us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:51.405197 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:51.405901 (+ 704us) server_negotiation.cc:299] Negotiation successful
0809 22:37:51.406355 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":336,"threads_started":1}
I20260809 22:37:51.408532 6116 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:51.409055 6116 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:37:51.409183 6116 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cedb8a951644df9ee18e56b11d9cd5, State: Initialized, Role: FOLLOWER
I20260809 22:37:51.409861 6116 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:51.410166 6116 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:51.410321 6116 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:51.410516 6116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:37:51.412823 6116 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:51.413348 6116 leader_election.cc:302] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cedb8a951644df9ee18e56b11d9cd5; no voters:
I20260809 22:37:51.413988 6116 leader_election.cc:288] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:37:51.414140 6121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:37:51.414964 6121 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 LEADER]: Becoming Leader. State: Replica: a1cedb8a951644df9ee18e56b11d9cd5, State: Running, Role: LEADER
I20260809 22:37:51.415539 6121 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:51.416213 6116 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:51.418926 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } } }
I20260809 22:37:51.418900 6123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cedb8a951644df9ee18e56b11d9cd5. Latest consensus state: current_term: 1 leader_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } } }
I20260809 22:37:51.419445 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:51.419445 6123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:51.420491 6131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:51.427253 6131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:51.437320 6131 catalog_manager.cc:1347] Generated new cluster ID: 6b8dedbf52024d93931c8de45757a932
I20260809 22:37:51.437497 6131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:51.450708 6131 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:37:51.452107 6131 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:51.459877 6131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Generated new TSK 0
I20260809 22:37:51.460840 6131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:37:51.544929 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41471
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:51.914382 6144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:51.914695 6144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:51.914795 6144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:51.923100 6144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:51.923228 6144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:51.923344 6144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:51.933686 6144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:51.935074 6144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:51.936359 6144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:51.950325 6149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:51.950248 6153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:51.950282 6150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:51.951248 6144 server_base.cc:1034] running on GCE node
I20260809 22:37:51.951897 6144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:51.953042 6144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:51.954308 6144 hybrid_clock.cc:648] HybridClock initialized: now 1786315071954232 us; error 69 us; skew 500 ppm
I20260809 22:37:51.954716 6144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:51 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315071, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:51.958832 6144 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:37:51.960819 6144 webserver.cc:460] Webserver started at http://127.28.243.65:39745/ using document root <none> and password file <none>
I20260809 22:37:51.961580 6144 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:51.961730 6144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:51.962067 6144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:51.965312 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "c70ca67b23b64e538d0f8d4ab1700678"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "345e94c29b595a891a12fda78e150583"
server_key_iv: "f2f3bc7bcbaff12bacca3acc6b2eb774"
server_key_version: "encryptionkey@0"
I20260809 22:37:51.966176 6144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "c70ca67b23b64e538d0f8d4ab1700678"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "345e94c29b595a891a12fda78e150583"
server_key_iv: "f2f3bc7bcbaff12bacca3acc6b2eb774"
server_key_version: "encryptionkey@0"
I20260809 22:37:51.971814 6144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 22:37:51.975695 6162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:51.977648 6144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:51.977862 6144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "c70ca67b23b64e538d0f8d4ab1700678"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "345e94c29b595a891a12fda78e150583"
server_key_iv: "f2f3bc7bcbaff12bacca3acc6b2eb774"
server_key_version: "encryptionkey@0"
I20260809 22:37:51.978133 6144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:51.999876 6144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:52.005546 6144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:52.005915 6144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:52.007063 6144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:52.008630 6144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:52.008744 6144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:52.008913 6144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:52.008987 6144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:52.047796 6144 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:41157
I20260809 22:37:52.047832 6281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:41157 every 8 connection(s)
I20260809 22:37:52.049194 6144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:52.049983 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6144
I20260809 22:37:52.050225 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:37:52.051352 29645 external_mini_cluster.cc:1428] Setting key 1e74bee8b17370a33038d78da43f2fa9
Aug 09 22:37:52 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315071, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:52.077373 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:52.052209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:41067 (local address 127.28.243.126:41471)
0809 22:37:52.052965 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:52.052969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:52.053957 (+ 988us) server_negotiation.cc:407] Connection header received
0809 22:37:52.056949 (+ 2992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:52.056953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:52.057102 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:52.057219 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:52.059522 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.060290 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.062884 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.063314 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.063427 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:52.067958 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:52.067968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:52.067976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:52.068025 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:52.072271 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:52.073448 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:52.073457 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:52.073464 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:52.073681 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:52.074518 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:52.074527 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:52.074533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:52.075119 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:52.075362 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:52.076416 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0809 22:37:52.076775 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":504,"threads_started":1}
I20260809 22:37:52.078163 6285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:52.052755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:41067)
0809 22:37:52.053576 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:52.053634 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:52.056589 (+ 2955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:52.057521 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:52.057535 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:52.058388 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:52.059284 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:52.059312 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.060463 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.060471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:52.062711 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:52.062730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.063574 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.063584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:52.063932 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:52.064844 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:52.064888 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:52.067695 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:52.072483 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:52.072495 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:52.072511 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:52.073262 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:52.073834 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:52.073838 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:52.073841 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:52.074254 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:52.075313 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:52.075322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:52.075896 (+ 574us) client_negotiation.cc:769] Sending connection context
0809 22:37:52.076234 (+ 338us) client_negotiation.cc:240] Negotiation successful
0809 22:37:52.076836 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":329,"threads_started":1}
I20260809 22:37:52.080158 6282 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41471
I20260809 22:37:52.080494 6282 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:52.081332 6282 heartbeater.cc:511] Master 127.28.243.126:41471 requested a full tablet report, sending...
I20260809 22:37:52.083812 6079 ts_manager.cc:194] Registered new tserver with Master: c70ca67b23b64e538d0f8d4ab1700678 (127.28.243.65:41157)
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:37:52.085410 6079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:41067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:37:52.125779 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41471
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:52.504376 6288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:52.504810 6288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:52.505004 6288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:52.512691 6288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:52.512821 6288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:52.513042 6288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:52.523954 6288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:52.525426 6288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:52.526865 6288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:52.541180 6297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:52.541157 6295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:52.541430 6294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:52.542641 6288 server_base.cc:1034] running on GCE node
I20260809 22:37:52.543406 6288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:52.544445 6288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:52.545710 6288 hybrid_clock.cc:648] HybridClock initialized: now 1786315072545677 us; error 101 us; skew 500 ppm
I20260809 22:37:52.546093 6288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:52 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:52.549877 6288 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:37:52.551846 6288 webserver.cc:460] Webserver started at http://127.28.243.66:35661/ using document root <none> and password file <none>
I20260809 22:37:52.552754 6288 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:52.552943 6288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:52.553337 6288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:52.556972 6288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "b9d2a9d506ee4bb9b2f1abb2cb73e937"
format_stamp: "Formatted at 2026-08-09 22:37:52 on dist-test-slave-rgb1"
server_key: "34d445c281197e45930185431aecb7a9"
server_key_iv: "2e58f22f0fae270717a8242c7b507109"
server_key_version: "encryptionkey@0"
I20260809 22:37:52.557996 6288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "b9d2a9d506ee4bb9b2f1abb2cb73e937"
format_stamp: "Formatted at 2026-08-09 22:37:52 on dist-test-slave-rgb1"
server_key: "34d445c281197e45930185431aecb7a9"
server_key_iv: "2e58f22f0fae270717a8242c7b507109"
server_key_version: "encryptionkey@0"
I20260809 22:37:52.564255 6288 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 22:37:52.568141 6306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:52.569785 6288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:52.570003 6288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "b9d2a9d506ee4bb9b2f1abb2cb73e937"
format_stamp: "Formatted at 2026-08-09 22:37:52 on dist-test-slave-rgb1"
server_key: "34d445c281197e45930185431aecb7a9"
server_key_iv: "2e58f22f0fae270717a8242c7b507109"
server_key_version: "encryptionkey@0"
I20260809 22:37:52.570248 6288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:52.589859 6288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:52.595075 6288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:52.595450 6288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:52.596772 6288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:52.598344 6288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:52.598459 6288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:52.598611 6288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:52.598695 6288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:52.637599 6288 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39003
I20260809 22:37:52.637651 6430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39003 every 8 connection(s)
I20260809 22:37:52.639027 6288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:52.648615 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6288
I20260809 22:37:52.648844 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:37:52.649343 29645 external_mini_cluster.cc:1428] Setting key 1efe6fe8ab33546fb92baf6930c69d83
Aug 09 22:37:52 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315072, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:52.664781 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:52.642563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:49655 (local address 127.28.243.126:41471)
0809 22:37:52.643124 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:52.643131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:52.644174 (+ 1043us) server_negotiation.cc:407] Connection header received
0809 22:37:52.646230 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:52.646235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:52.646317 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:52.646442 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:52.648264 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.649426 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.651727 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.652231 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.652338 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:52.657639 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:52.657646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:52.657650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:52.657692 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:52.660828 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:52.661760 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:52.661766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:52.661770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:52.661931 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:52.662467 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:52.662474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:52.662477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:52.662823 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:52.663063 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:52.663996 (+ 933us) server_negotiation.cc:299] Negotiation successful
0809 22:37:52.664278 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":255,"threads_started":1}
I20260809 22:37:52.665867 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:52.643135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:49655)
0809 22:37:52.643981 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:52.644030 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:52.645908 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:52.646726 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:52.646735 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:52.647315 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:52.648006 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:52.648027 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.649662 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.649668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:52.651560 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:52.651574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:52.652509 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:52.652521 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:52.652957 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:52.653830 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:52.653871 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:52.657292 (+ 3421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:52.661044 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:52.661056 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:52.661075 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:52.661615 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:52.662074 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:52.662080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:52.662084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:52.662329 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:52.663030 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:52.663039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:52.663566 (+ 527us) client_negotiation.cc:769] Sending connection context
0809 22:37:52.663889 (+ 323us) client_negotiation.cc:240] Negotiation successful
0809 22:37:52.664547 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":340,"threads_started":1}
I20260809 22:37:52.667596 6432 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41471
I20260809 22:37:52.667945 6432 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:52.668975 6432 heartbeater.cc:511] Master 127.28.243.126:41471 requested a full tablet report, sending...
I20260809 22:37:52.670845 6079 ts_manager.cc:194] Registered new tserver with Master: b9d2a9d506ee4bb9b2f1abb2cb73e937 (127.28.243.66:39003)
I20260809 22:37:52.671804 6079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:49655
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:37:52.717541 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41471
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:53.077307 6438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:53.077618 6438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:53.077782 6438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:53.085599 6438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:53.085728 6438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:53.085932 6438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:53.088814 6282 heartbeater.cc:503] Master 127.28.243.126:41471 was elected leader, sending a full tablet report...
I20260809 22:37:53.097235 6438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:53.098634 6438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:53.099972 6438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:53.113776 6443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:53.113731 6444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:53.113799 6446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:53.115522 6438 server_base.cc:1034] running on GCE node
I20260809 22:37:53.116319 6438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:53.117547 6438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:53.118793 6438 hybrid_clock.cc:648] HybridClock initialized: now 1786315073118747 us; error 41 us; skew 500 ppm
I20260809 22:37:53.119237 6438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:53 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:53.123246 6438 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:37:53.124998 6438 webserver.cc:460] Webserver started at http://127.28.243.67:41629/ using document root <none> and password file <none>
I20260809 22:37:53.125743 6438 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:53.125856 6438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:53.126327 6438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:37:53.129671 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "8ded1ef8ca6f4a9990d9d72d4c86d23a"
format_stamp: "Formatted at 2026-08-09 22:37:53 on dist-test-slave-rgb1"
server_key: "5c259e740af7b1479f7df4583d5be951"
server_key_iv: "8694c59ae1daa99a6ffd71aef49713b8"
server_key_version: "encryptionkey@0"
I20260809 22:37:53.130553 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "8ded1ef8ca6f4a9990d9d72d4c86d23a"
format_stamp: "Formatted at 2026-08-09 22:37:53 on dist-test-slave-rgb1"
server_key: "5c259e740af7b1479f7df4583d5be951"
server_key_iv: "8694c59ae1daa99a6ffd71aef49713b8"
server_key_version: "encryptionkey@0"
I20260809 22:37:53.137250 6438 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:37:53.141414 6455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:53.143389 6438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:37:53.143597 6438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "8ded1ef8ca6f4a9990d9d72d4c86d23a"
format_stamp: "Formatted at 2026-08-09 22:37:53 on dist-test-slave-rgb1"
server_key: "5c259e740af7b1479f7df4583d5be951"
server_key_iv: "8694c59ae1daa99a6ffd71aef49713b8"
server_key_version: "encryptionkey@0"
I20260809 22:37:53.143874 6438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:53.161669 6438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:53.167066 6438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:53.167435 6438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:53.168610 6438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:53.170095 6438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:53.170183 6438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:53.170316 6438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:53.170373 6438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:53.210757 6438 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:34917
I20260809 22:37:53.210841 6576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:34917 every 8 connection(s)
I20260809 22:37:53.212226 6438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:53.218514 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6438
I20260809 22:37:53.218706 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:37:53.219319 29645 external_mini_cluster.cc:1428] Setting key 760fb45e20dd9b6db557de721771c37b
Aug 09 22:37:53 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315073, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:37:53.237157 6580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:53.215197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:36725 (local address 127.28.243.126:41471)
0809 22:37:53.215766 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:53.215774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:53.216906 (+ 1132us) server_negotiation.cc:407] Connection header received
0809 22:37:53.219463 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:53.219468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:53.219642 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:53.219758 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:53.222030 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.223363 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.225123 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.225564 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.225675 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:53.229790 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:53.229796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:53.229799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:53.229847 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:53.233035 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.233858 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.233863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.233866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.234088 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.234883 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.234889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.234891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.235287 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:53.235465 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:53.236519 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0809 22:37:53.236806 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260809 22:37:53.238220 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:53.215732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:36725)
0809 22:37:53.216733 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:53.216771 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:53.219064 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:53.220066 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:53.220093 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:53.220817 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:53.221711 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:53.221738 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.223512 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.223516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:53.224974 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:53.224987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.225804 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.225810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:53.226101 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:53.226911 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:53.226937 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:53.229525 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:37:53.233207 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:53.233215 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:53.233228 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:53.233681 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:53.234281 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:37:53.234287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:37:53.234291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:37:53.234666 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:37:53.235470 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:37:53.235479 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:37:53.236042 (+ 563us) client_negotiation.cc:769] Sending connection context
0809 22:37:53.236338 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 22:37:53.236868 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":343,"threads_started":1}
I20260809 22:37:53.239755 6577 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41471
I20260809 22:37:53.240070 6577 heartbeater.cc:464] Registering TS with master...
I20260809 22:37:53.241043 6577 heartbeater.cc:511] Master 127.28.243.126:41471 requested a full tablet report, sending...
I20260809 22:37:53.242780 6079 ts_manager.cc:194] Registered new tserver with Master: 8ded1ef8ca6f4a9990d9d72d4c86d23a (127.28.243.67:34917)
I20260809 22:37:53.243701 6079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:36725
I20260809 22:37:53.248872 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab.
Entry for principal oryx/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab.
WARNING: no policy specified for oryx/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab.
Entry for principal oryx/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab.
WARNING: no policy specified for oryx/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab.
Entry for principal oryx/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab.
I20260809 22:37:53.441375 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6144
Aug 09 22:37:53 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:37:53.460670 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:53.442762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.28.243.65:41157)
0809 22:37:53.443636 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:53.443644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:53.443671 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:37:53.443733 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:53.443739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:53.443955 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:53.444110 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:53.445765 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.446871 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.448433 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.449234 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.449407 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:53.454257 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:53.454287 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:53.454305 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:53.454350 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:53.457612 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.458450 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.458459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.458462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.458660 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.459063 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.459067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.459069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.459454 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:53.459737 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:53.460039 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 22:37:53.460250 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":506,"threads_started":1}
I20260809 22:37:53.462353 6277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:53.550817 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6144
I20260809 22:37:53.571148 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6288
Aug 09 22:37:53 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:37:53.592089 6592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:53.572446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.28.243.66:39003)
0809 22:37:53.573551 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:53.573560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:53.573591 (+ 31us) server_negotiation.cc:407] Connection header received
0809 22:37:53.573659 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:53.573667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:53.573911 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:53.574077 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:53.575967 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.577456 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.579172 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.579916 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.580079 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:53.585392 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:53.585411 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:53.585427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:53.585471 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:53.588694 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.589565 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.589572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.589574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.589711 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.590165 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.590169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.590171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.590577 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:53.590830 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:53.591370 (+ 540us) server_negotiation.cc:299] Negotiation successful
0809 22:37:53.591684 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":778,"threads_started":1}
I20260809 22:37:53.593684 6426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:53.679153 6432 heartbeater.cc:503] Master 127.28.243.126:41471 was elected leader, sending a full tablet report...
I20260809 22:37:53.681003 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6288
I20260809 22:37:53.699411 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6438
Aug 09 22:37:53 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:37:53.718070 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:53.700532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.28.243.67:34917)
0809 22:37:53.701136 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:53.701144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:53.701174 (+ 30us) server_negotiation.cc:407] Connection header received
0809 22:37:53.701233 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:53.701239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:53.701451 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:53.701581 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:53.703123 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.704320 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.705936 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:53.706556 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:53.706672 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:53.711420 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:53.711434 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:53.711449 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:53.711485 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:53.714657 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.715376 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.715383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.715385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.715508 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:53.715825 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:53.715828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:53.715830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:53.716268 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:53.716548 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:53.717357 (+ 809us) server_negotiation.cc:299] Negotiation successful
0809 22:37:53.717689 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":223,"threads_started":1}
I20260809 22:37:53.719437 6572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:53.803897 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6438
I20260809 22:37:53.822042 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6045
I20260809 22:37:53.823407 6111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:37:53.930645 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6045
I20260809 22:37:53.953924 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41471
--webserver_interface=127.28.243.126
--webserver_port=33805
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:37:54.316217 6597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:54.316619 6597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:54.316819 6597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:54.324729 6597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:37:54.324867 6597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:54.325062 6597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:54.325156 6597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:37:54.325251 6597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:37:54.335939 6597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=33805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:54.337385 6597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:54.338780 6597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:54.351451 6602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:54.353086 6603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:54.353221 6606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:54.353180 6597 server_base.cc:1034] running on GCE node
I20260809 22:37:54.354185 6597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:54.355690 6597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:54.356884 6597 hybrid_clock.cc:648] HybridClock initialized: now 1786315074356811 us; error 71 us; skew 500 ppm
I20260809 22:37:54.357308 6597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:54 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315074, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:54.365195 6597 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:37:54.367254 6597 webserver.cc:460] Webserver started at http://127.28.243.126:33805/ using document root <none> and password file <none>
I20260809 22:37:54.368142 6597 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:54.368320 6597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:54.374778 6597 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 22:37:54.378558 6614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:54.380329 6597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260809 22:37:54.380640 6597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "a1cedb8a951644df9ee18e56b11d9cd5"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "373f3f95e4ec14f1b71cc40d2d7d463b"
server_key_iv: "e0c371beb1931781b87a9602f507aac4"
server_key_version: "encryptionkey@0"
I20260809 22:37:54.381950 6597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:54.405359 6597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:54.410812 6597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:54.411164 6597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:54.434441 6597 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:41471
I20260809 22:37:54.434499 6668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:41471 every 8 connection(s)
I20260809 22:37:54.437088 6597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:37:54.445727 6669 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:37:54.446009 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6597
I20260809 22:37:54.447268 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:41157
--local_ip_for_outbound_sockets=127.28.243.65
--tserver_master_addrs=127.28.243.126:41471
--webserver_port=39745
--webserver_interface=127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:37:54.451359 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Bootstrap starting.
I20260809 22:37:54.474505 6669 log.cc:824] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Log is configured to *not* fsync() on all Append() calls
I20260809 22:37:54.486003 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:37:54.486745 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Bootstrap complete.
I20260809 22:37:54.498291 6669 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:54.499305 6669 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cedb8a951644df9ee18e56b11d9cd5, State: Initialized, Role: FOLLOWER
I20260809 22:37:54.500043 6669 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:54.500334 6669 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:37:54.500488 6669 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:37:54.500787 6669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:37:54.505733 6669 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:54.506799 6669 leader_election.cc:302] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cedb8a951644df9ee18e56b11d9cd5; no voters:
I20260809 22:37:54.507763 6669 leader_election.cc:288] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 22:37:54.507928 6674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:37:54.509850 6674 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [term 2 LEADER]: Becoming Leader. State: Replica: a1cedb8a951644df9ee18e56b11d9cd5, State: Running, Role: LEADER
I20260809 22:37:54.510473 6674 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } }
I20260809 22:37:54.510692 6669 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:37:54.513844 6677 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cedb8a951644df9ee18e56b11d9cd5. Latest consensus state: current_term: 2 leader_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } } }
I20260809 22:37:54.514323 6677 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:54.514843 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cedb8a951644df9ee18e56b11d9cd5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41471 } } }
I20260809 22:37:54.515228 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5 [sys.catalog]: This master's current role is: LEADER
I20260809 22:37:54.517668 6682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:37:54.525367 6682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:37:54.527361 6682 catalog_manager.cc:1259] Loaded cluster ID: 6b8dedbf52024d93931c8de45757a932
I20260809 22:37:54.527514 6682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:37:54.532902 6682 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:37:54.535010 6682 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1cedb8a951644df9ee18e56b11d9cd5: Loaded TSK: 0
I20260809 22:37:54.537042 6682 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 22:37:54.859138 6672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:54.859495 6672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:54.859663 6672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:54.868427 6672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:54.868747 6672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:54.868947 6672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:37:54.882467 6672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.65.keytab
--principal=oryx/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:41157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=39745
--tserver_master_addrs=127.28.243.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:54.883994 6672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:54.885550 6672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:54.902920 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:54.903240 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:54.903471 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:54.903818 6672 server_base.cc:1034] running on GCE node
I20260809 22:37:54.904721 6672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:54.905820 6672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:54.907142 6672 hybrid_clock.cc:648] HybridClock initialized: now 1786315074907051 us; error 82 us; skew 500 ppm
I20260809 22:37:54.907599 6672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:54 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315074, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:54.911995 6672 init.cc:377] Logged in from keytab as oryx/127.28.243.65@KRBTEST.COM (short username oryx)
I20260809 22:37:54.914170 6672 webserver.cc:460] Webserver started at http://127.28.243.65:39745/ using document root <none> and password file <none>
I20260809 22:37:54.915230 6672 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:54.915403 6672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:54.923280 6672 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:37:54.927402 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:54.929315 6672 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:37:54.929522 6672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "c70ca67b23b64e538d0f8d4ab1700678"
format_stamp: "Formatted at 2026-08-09 22:37:51 on dist-test-slave-rgb1"
server_key: "345e94c29b595a891a12fda78e150583"
server_key_iv: "f2f3bc7bcbaff12bacca3acc6b2eb774"
server_key_version: "encryptionkey@0"
I20260809 22:37:54.930943 6672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:54.973544 6672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:54.979979 6672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:54.980348 6672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:54.981554 6672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:54.982985 6672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:54.983091 6672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:54.983167 6672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:54.983227 6672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:55.023674 6672 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:41157
I20260809 22:37:55.023834 6832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:41157 every 8 connection(s)
I20260809 22:37:55.025341 6672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:37:55.032675 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6672
I20260809 22:37:55.033447 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:39003
--local_ip_for_outbound_sockets=127.28.243.66
--tserver_master_addrs=127.28.243.126:41471
--webserver_port=35661
--webserver_interface=127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:55.048391 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:55.029038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:44241)
0809 22:37:55.029937 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:55.030006 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:55.032040 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:55.033668 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:55.033681 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:55.034597 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:55.035501 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.035534 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.037680 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.037689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.040072 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.040093 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.041418 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.041428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.041788 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.042729 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:55.042767 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:55.047373 (+ 4606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":413,"threads_started":1}
W20260809 22:37:55.049137 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:55.028921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:44241 (local address 127.28.243.126:41471)
0809 22:37:55.029646 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:55.029658 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:55.030177 (+ 519us) server_negotiation.cc:407] Connection header received
0809 22:37:55.032706 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:55.032718 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:55.033112 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:55.033411 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:55.035916 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.037480 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.040285 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.041151 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.041303 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.047827 (+ 6524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:44241: BlockingRecv error: recv got EOF from 127.28.243.65:44241 (error 108)
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":297,"threads_started":1}
W20260809 22:37:55.049286 6833 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:41471 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.243.126:41471: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:55.058379 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:55.049925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:55171)
0809 22:37:55.050146 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:55.050170 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:55.050288 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:55.050624 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:55.050629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:55.051001 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:55.051145 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.051154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.052195 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.052200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.053170 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.053181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.053879 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.053885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.054062 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.054689 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:55.054707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:55.058085 (+ 3378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 22:37:55.058610 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:55.050085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:55171 (local address 127.28.243.126:41471)
0809 22:37:55.050288 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:55.050295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:55.050314 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:37:55.050407 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:55.050412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:55.050486 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:55.050599 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:55.051295 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.052049 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.053325 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.053730 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.053867 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.058371 (+ 4504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:55171: BlockingRecv error: recv got EOF from 127.28.243.65:55171 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:55.067674 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:55.059140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:47685)
0809 22:37:55.059466 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:55.059486 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:55.059606 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:55.059938 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:55.059942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:55.060266 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:55.060412 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.060420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.061465 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.061471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.062405 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.062416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.063128 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.063134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.063324 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.064181 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:55.064200 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:55.067335 (+ 3135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226}
W20260809 22:37:55.068107 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:55.059299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:47685 (local address 127.28.243.126:41471)
0809 22:37:55.059462 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:55.059468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:55.059538 (+ 70us) server_negotiation.cc:407] Connection header received
0809 22:37:55.059746 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:55.059750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:55.059807 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:55.059910 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:55.060600 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.061315 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.062550 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.062969 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.063112 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.067583 (+ 4471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:47685: BlockingRecv error: recv got EOF from 127.28.243.65:47685 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260809 22:37:55.068125 6833 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 22:37:55.400385 6837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:55.400852 6837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:55.401031 6837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:55.413938 6837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:55.414103 6837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:55.414322 6837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:37:55.428318 6837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.66.keytab
--principal=oryx/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:39003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=35661
--tserver_master_addrs=127.28.243.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:55.429837 6837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:55.431214 6837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:55.447237 6843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:55.447471 6846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:55.447911 6844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:55.448926 6837 server_base.cc:1034] running on GCE node
I20260809 22:37:55.449640 6837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:55.450704 6837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:55.451956 6837 hybrid_clock.cc:648] HybridClock initialized: now 1786315075451916 us; error 66 us; skew 500 ppm
I20260809 22:37:55.452354 6837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:55.455925 6837 init.cc:377] Logged in from keytab as oryx/127.28.243.66@KRBTEST.COM (short username oryx)
I20260809 22:37:55.457618 6837 webserver.cc:460] Webserver started at http://127.28.243.66:35661/ using document root <none> and password file <none>
I20260809 22:37:55.458386 6837 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:55.458560 6837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:55.464468 6837 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260809 22:37:55.467636 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:55.469424 6837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:55.469624 6837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "b9d2a9d506ee4bb9b2f1abb2cb73e937"
format_stamp: "Formatted at 2026-08-09 22:37:52 on dist-test-slave-rgb1"
server_key: "34d445c281197e45930185431aecb7a9"
server_key_iv: "2e58f22f0fae270717a8242c7b507109"
server_key_version: "encryptionkey@0"
I20260809 22:37:55.470713 6837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:55.493468 6837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:55.499310 6837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:55.499634 6837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:55.500805 6837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:55.502354 6837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:55.502492 6837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:55.502638 6837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:55.502728 6837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:55.539194 6837 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39003
I20260809 22:37:55.539251 6980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39003 every 8 connection(s)
I20260809 22:37:55.540850 6837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:37:55.548447 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6837
I20260809 22:37:55.549235 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:34917
--local_ip_for_outbound_sockets=127.28.243.67
--tserver_master_addrs=127.28.243.126:41471
--webserver_port=41629
--webserver_interface=127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 22:37:55.557711 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:55.543667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:46233 (local address 127.28.243.126:41471)
0809 22:37:55.543847 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:55.543852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:55.545134 (+ 1282us) server_negotiation.cc:407] Connection header received
0809 22:37:55.547144 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:55.547149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:55.547244 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:55.547358 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:55.549217 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.550034 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.551644 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.552064 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.552206 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.557416 (+ 5210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:46233: BlockingRecv error: recv got EOF from 127.28.243.66:46233 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 22:37:55.558451 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:55.544160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:46233)
0809 22:37:55.544926 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:55.544964 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:55.546859 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:55.547601 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:55.547609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:55.548293 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:55.549023 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.549042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.550207 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.550211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.551477 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.551490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.552314 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.552321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.552625 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.553352 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:55.553381 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:55.557201 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":366,"threads_started":1}
W20260809 22:37:55.559262 6981 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:41471 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.243.126:41471: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:55.569754 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:55.559968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:51095)
0809 22:37:55.560212 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:55.560237 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:55.560370 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:55.560828 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:55.560834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:55.561241 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:55.561402 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.561411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.562630 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.562639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.563676 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.563689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.564514 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.564521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.564749 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.565530 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:55.565552 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:55.569343 (+ 3791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260809 22:37:55.571472 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:55.560168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:51095 (local address 127.28.243.126:41471)
0809 22:37:55.560363 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:55.560370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:55.560392 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:37:55.560546 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:55.560552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:55.560679 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:55.560799 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:55.561591 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.562469 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.563858 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.564335 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.564496 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.571140 (+ 6644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:51095: BlockingRecv error: recv got EOF from 127.28.243.66:51095 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:55.585366 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:55.570636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:59763)
0809 22:37:55.572045 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:55.572067 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:55.572196 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:55.572861 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:55.572868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:55.573619 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:55.573917 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.573936 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.576280 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.576288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.578435 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:55.578456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.579798 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.579807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:55.580050 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.580928 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:55.580960 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:55.585045 (+ 4085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1276}
W20260809 22:37:55.585763 6981 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 22:37:55.590911 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:55.570994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:59763 (local address 127.28.243.126:41471)
0809 22:37:55.571876 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:55.571885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:55.572315 (+ 430us) server_negotiation.cc:407] Connection header received
0809 22:37:55.572384 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:55.572392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:55.572517 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:55.572653 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:55.574164 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.576013 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.578672 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:55.579588 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:55.579767 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:55.590538 (+ 10771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:59763: BlockingRecv error: recv got EOF from 127.28.243.66:59763 (error 108)
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":5881,"threads_started":1}
W20260809 22:37:55.910709 6985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:37:55.911204 6985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:37:55.911408 6985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:37:55.919100 6985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:37:55.919221 6985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:37:55.919386 6985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:37:55.929576 6985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/krb5kdc/oryx_127.28.243.67.keytab
--principal=oryx/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:34917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=41629
--tserver_master_addrs=127.28.243.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:37:55.931391 6985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:37:55.933511 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:37:55.949434 6996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:55.949410 6992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:37:55.949635 6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:37:55.950736 6985 server_base.cc:1034] running on GCE node
I20260809 22:37:55.951704 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:37:55.953095 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:37:55.954365 6985 hybrid_clock.cc:648] HybridClock initialized: now 1786315075954294 us; error 65 us; skew 500 ppm
I20260809 22:37:55.954767 6985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:37:55 dist-test-slave-rgb1 krb5kdc[6030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315075, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:37:55.958333 6985 init.cc:377] Logged in from keytab as oryx/127.28.243.67@KRBTEST.COM (short username oryx)
I20260809 22:37:55.959916 6985 webserver.cc:460] Webserver started at http://127.28.243.67:41629/ using document root <none> and password file <none>
I20260809 22:37:55.960717 6985 fs_manager.cc:359] Metadata directory not provided
I20260809 22:37:55.960886 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:37:55.966804 6985 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:37:55.969887 7004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:55.971369 6985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:37:55.971583 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "8ded1ef8ca6f4a9990d9d72d4c86d23a"
format_stamp: "Formatted at 2026-08-09 22:37:53 on dist-test-slave-rgb1"
server_key: "5c259e740af7b1479f7df4583d5be951"
server_key_iv: "8694c59ae1daa99a6ffd71aef49713b8"
server_key_version: "encryptionkey@0"
I20260809 22:37:55.972630 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:37:55.986356 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:37:55.991041 6985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:37:55.991416 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:37:55.992640 6985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:37:55.994308 6985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:37:55.994460 6985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:55.994625 6985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:37:55.994738 6985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:37:56.038029 6985 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:34917
I20260809 22:37:56.038100 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:34917 every 8 connection(s)
I20260809 22:37:56.039641 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:37:56.043143 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 6985
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 22:37:56.058012 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:56.042777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:53017 (local address 127.28.243.126:41471)
0809 22:37:56.043023 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:56.043030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:56.044777 (+ 1747us) server_negotiation.cc:407] Connection header received
0809 22:37:56.047155 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:56.047160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:56.047287 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:56.047400 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:56.049122 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.049834 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.051369 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.051834 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.051977 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.057702 (+ 5725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:53017: BlockingRecv error: recv got EOF from 127.28.243.67:53017 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:37:56.058717 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:56.043415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:53017)
0809 22:37:56.044508 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:56.044591 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:56.046889 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:56.047716 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:56.047725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:56.048261 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:56.048945 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.048964 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.049992 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.049997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.051222 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.051234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.052079 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.052087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.052312 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.053081 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:56.053109 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:56.057499 (+ 4390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":358,"threads_started":1}
W20260809 22:37:56.059388 7130 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:41471 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.243.126:41471: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
I20260809 22:37:56.061213 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:56.044673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.28.243.126:41471)
0809 22:37:56.044950 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:56.044955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:56.045612 (+ 657us) server_negotiation.cc:407] Connection header received
0809 22:37:56.045773 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:56.045779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:56.045871 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:56.046006 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:56.047660 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.048523 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.049877 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.050297 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.050439 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.053390 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:37:56.053399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:37:56.053414 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:37:56.053449 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:37:56.057198 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:56.057974 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:56.057982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:56.057993 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:56.058258 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:37:56.058735 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:37:56.058741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:37:56.058745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:37:56.059482 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:37:56.059806 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 22:37:56.060328 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 22:37:56.060762 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:56.070906 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:56.060153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:58169)
0809 22:37:56.060369 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:56.060389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:56.060505 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:56.060986 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:56.060990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:56.061399 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:56.061586 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.061595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.063032 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.063037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.064465 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.064481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.065558 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.065564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.066025 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.066796 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:56.066814 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:56.070621 (+ 3807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 22:37:56.071780 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:56.060470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:58169 (local address 127.28.243.126:41471)
0809 22:37:56.060686 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:56.060693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:56.060713 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:56.060762 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:56.060766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:56.060836 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:56.060947 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:56.061740 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.062865 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.064754 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.065383 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.065494 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.070898 (+ 5404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:58169: BlockingRecv error: recv got EOF from 127.28.243.67:58169 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:56.083120 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:56.068930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:53127)
0809 22:37:56.071527 (+ 2597us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:56.071572 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:56.071735 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:56.072685 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:56.072690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:56.073361 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:56.073634 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.073653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.074889 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.074895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.076783 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.076804 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.078064 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.078073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.078290 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.079040 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:56.079070 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:56.082857 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2441,"thread_start_us":2227,"threads_started":1}
I20260809 22:37:56.083230 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:56.071883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:33545)
0809 22:37:56.072165 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:56.072189 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:56.072313 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:56.072833 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:56.072842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:56.073283 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:56.073520 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.073536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.075339 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.075345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.077117 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.077134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.078302 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.078308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.078549 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.079207 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:56.079226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:56.082909 (+ 3683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260809 22:37:56.083459 6986 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:56.069047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:53127 (local address 127.28.243.126:41471)
0809 22:37:56.069360 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:56.069364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:56.071686 (+ 2322us) server_negotiation.cc:407] Connection header received
0809 22:37:56.072129 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:56.072134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:56.072256 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:56.072379 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:56.073812 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.074716 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.076975 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.077805 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.077960 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.083171 (+ 5211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:53127: BlockingRecv error: recv got EOF from 127.28.243.65:53127 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":164}
W20260809 22:37:56.083564 7130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 22:37:56.084017 6835 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:56.072028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:33545 (local address 127.28.243.126:41471)
0809 22:37:56.072303 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:56.072309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:56.072328 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:37:56.072469 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:56.072475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:56.072650 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:56.072789 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:56.073814 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.075022 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.077304 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.078096 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.078261 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.083800 (+ 5539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:33545: BlockingRecv error: recv got EOF from 127.28.243.67:33545 (error 108)
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":170,"thread_start_us":585,"threads_started":1}
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:56 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:56.596525 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:56.586514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:51131)
0809 22:37:56.587076 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:56.587093 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:56.587218 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:56.587602 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:56.587606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:56.588124 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:56.588333 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.588348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.589653 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.589659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.590809 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:56.590824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.591611 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.591618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:56.591802 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.592685 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:56.592713 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:56.596262 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
W20260809 22:37:56.596824 7137 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:56.586601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:51131 (local address 127.28.243.126:41471)
0809 22:37:56.587179 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:56.587184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:56.587204 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:56.587340 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:56.587344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:56.587412 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:56.587516 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:56.588541 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.589455 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.590963 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:56.591380 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:56.591483 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:56.596487 (+ 5004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:51131: BlockingRecv error: recv got EOF from 127.28.243.66:51131 (error 108)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":361,"threads_started":1}
Aug 09 22:37:57 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:57 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:57 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:57 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:57.095110 7139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:57.084337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:36209)
0809 22:37:57.085581 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:57.085601 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:57.085736 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:57.086094 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:57.086102 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:57.086665 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:57.086908 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:57.086928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.088168 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.088174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:57.089275 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:57.089311 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.090337 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.090344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:57.090570 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:57.091309 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:57.091333 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:57.094784 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":960,"threads_started":1}
I20260809 22:37:57.095345 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:57.084312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:58235)
0809 22:37:57.085215 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:57.085241 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:57.085389 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:57.086278 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:57.086285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:57.086737 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:57.087031 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:57.087047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.088329 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.088334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:57.089459 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:57.089475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.090538 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.090545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:57.090730 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:57.091551 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:57.091574 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:57.095019 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":672,"threads_started":1}
W20260809 22:37:57.095404 7137 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:57.084448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:36209 (local address 127.28.243.126:41471)
0809 22:37:57.084781 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:57.084790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:57.085703 (+ 913us) server_negotiation.cc:407] Connection header received
0809 22:37:57.085862 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:57.085866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:57.085937 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:57.086028 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:57.087063 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.087981 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.089459 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.090125 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.090293 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:57.095055 (+ 4762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:36209: BlockingRecv error: recv got EOF from 127.28.243.67:36209 (error 108)
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":154}
W20260809 22:37:57.095788 7140 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:57.084664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:58235 (local address 127.28.243.126:41471)
0809 22:37:57.085917 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:57.085921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:57.085934 (+ 13us) server_negotiation.cc:407] Connection header received
0809 22:37:57.085973 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:57.085976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:57.086065 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:57.086173 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:57.087210 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.088148 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.089633 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.090323 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.091078 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:57.095366 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:58235: BlockingRecv error: recv got EOF from 127.28.243.65:58235 (error 108)
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":1132,"thread_start_us":964,"threads_started":1}
Aug 09 22:37:57 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:57 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:57.607717 7141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:57.597708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:58567)
0809 22:37:57.598290 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:57.598308 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:57.598445 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:57.598963 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:57.598967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:57.599419 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:57.599628 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:57.599638 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.600991 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.600997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:57.602038 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:57.602052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.603090 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.603097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:57.603288 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:57.604028 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:57.604049 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:57.607326 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":335,"threads_started":1}
W20260809 22:37:57.607852 7142 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:57.597886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:58567 (local address 127.28.243.126:41471)
0809 22:37:57.598654 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:57.598662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:57.598680 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:37:57.598729 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:57.598733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:57.598793 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:57.598896 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:57.599786 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.600822 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.602237 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:57.602895 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:57.603065 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:57.607573 (+ 4508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:58567: BlockingRecv error: recv got EOF from 127.28.243.66:58567 (error 108)
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":535,"threads_started":1}
Aug 09 22:37:58 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:58 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:58 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:58 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:58.106668 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:58.096297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:35669)
0809 22:37:58.096891 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:58.096913 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:58.097170 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:58.097790 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:58.097796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:58.098394 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:58.098637 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:58.098649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.099822 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.099829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:58.100889 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:58.100902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.101784 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.101790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:58.101970 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:58.102646 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:58.102664 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:58.106315 (+ 3651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":415,"threads_started":1}
W20260809 22:37:58.106853 7142 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:58.096483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:35669 (local address 127.28.243.126:41471)
0809 22:37:58.096715 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:58.096722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:58.097074 (+ 352us) server_negotiation.cc:407] Connection header received
0809 22:37:58.097308 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:58.097316 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:58.097414 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:58.097555 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:58.098790 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.099675 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.101044 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.101581 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.101704 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:58.106581 (+ 4877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:35669: BlockingRecv error: recv got EOF from 127.28.243.67:35669 (error 108)
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":51}
I20260809 22:37:58.107861 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:58.096311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:56709)
0809 22:37:58.097965 (+ 1654us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:58.097987 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:58.098110 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:58.098596 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:58.098602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:58.098988 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:58.099152 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:58.099161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.100398 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.100406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:58.101552 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:58.101564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.102476 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.102482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:58.102682 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:58.103255 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:58.103276 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:58.107603 (+ 4327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1491,"thread_start_us":1366,"threads_started":1}
W20260809 22:37:58.108319 7146 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:58.096655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:56709 (local address 127.28.243.126:41471)
0809 22:37:58.098269 (+ 1614us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:58.098275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:58.098291 (+ 16us) server_negotiation.cc:407] Connection header received
0809 22:37:58.098334 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:58.098349 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:58.098427 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:58.098550 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:58.099316 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.100232 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.101700 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.102321 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.102458 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:58.107855 (+ 5397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:56709: BlockingRecv error: recv got EOF from 127.28.243.65:56709 (error 108)
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":1512,"thread_start_us":1382,"threads_started":1}
Aug 09 22:37:58 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:58 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:58.618312 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:58.608966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:35655)
0809 22:37:58.609531 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:58.609552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:58.609678 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:58.610068 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:58.610072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:58.610505 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:58.610676 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:58.610685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.611962 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.611966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:58.613120 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:58.613143 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.614205 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.614214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:58.614381 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:58.615080 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:58.615099 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:58.618054 (+ 2955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
W20260809 22:37:58.618609 7149 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:58.609174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:35655 (local address 127.28.243.126:41471)
0809 22:37:58.609665 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:58.609670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:58.609690 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:37:58.609807 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:58.609812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:58.609880 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:58.609979 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:58.610833 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.611763 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.613271 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:58.614008 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:58.614149 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:58.618289 (+ 4140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:35655: BlockingRecv error: recv got EOF from 127.28.243.66:35655 (error 108)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
Aug 09 22:37:59 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:59 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:59 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:59 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:59.121114 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:59.109069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:48445)
0809 22:37:59.110169 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:59.110192 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:59.110320 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:59.110811 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:59.110816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:59.111209 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:59.111354 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:59.111363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.113089 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.113097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:59.114802 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:59.114825 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.116121 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.116126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:59.116252 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:59.116938 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:59.116956 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:59.120755 (+ 3799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":747,"threads_started":1}
I20260809 22:37:59.121234 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:59.108004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:33683)
0809 22:37:59.108868 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:59.108890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:59.109040 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:59.109533 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:59.109542 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:59.110190 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:59.110458 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:59.110474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.111828 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.111835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:59.113573 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:59.113596 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.114696 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.114703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:59.115023 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:59.115976 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:59.116012 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:59.120755 (+ 4743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":465,"threads_started":1}
W20260809 22:37:59.121486 7149 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:59.108290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:33683 (local address 127.28.243.126:41471)
0809 22:37:59.108682 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:59.108689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:59.108987 (+ 298us) server_negotiation.cc:407] Connection header received
0809 22:37:59.109194 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:59.109202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:59.109300 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:59.109457 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:59.110632 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.111644 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.113772 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.114519 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.114668 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:59.121032 (+ 6364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:33683: BlockingRecv error: recv got EOF from 127.28.243.67:33683 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
W20260809 22:37:59.122972 7153 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:59.109149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:48445 (local address 127.28.243.126:41471)
0809 22:37:59.110486 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:59.110492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:59.110511 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:37:59.110567 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:59.110575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:59.110654 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:59.110778 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:59.111592 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.112869 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.115013 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.115941 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.116102 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:59.122476 (+ 6374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:48445: BlockingRecv error: recv got EOF from 127.28.243.65:48445 (error 108)
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":933,"threads_started":1}
Aug 09 22:37:59 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:37:59 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:37:59.635017 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:37:59.619638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:46041)
0809 22:37:59.620424 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0809 22:37:59.620440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:37:59.620609 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:37:59.621042 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:37:59.621048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:37:59.621636 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:37:59.621794 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:59.621803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.622987 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.622995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:59.623971 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:37:59.623982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.624772 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.624778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:37:59.624966 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:59.630426 (+ 5460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:37:59.630465 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:37:59.634585 (+ 4120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":534,"threads_started":1}
W20260809 22:37:59.635144 7153 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:37:59.619734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:46041 (local address 127.28.243.126:41471)
0809 22:37:59.620058 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0809 22:37:59.620070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:37:59.620541 (+ 471us) server_negotiation.cc:407] Connection header received
0809 22:37:59.620760 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:37:59.620765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:37:59.620855 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:37:59.621009 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:37:59.621966 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.622847 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.624123 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:37:59.624607 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:37:59.624745 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:37:59.634834 (+ 10089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:46041: BlockingRecv error: recv got EOF from 127.28.243.66:46041 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 22:38:00 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:00 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:00 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:00 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:00.133950 7155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:00.122280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:49093)
0809 22:38:00.122913 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:00.122937 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:00.123101 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:00.123582 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:00.123585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:00.123959 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:00.124312 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:00.124328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.126050 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.126056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:00.127302 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:00.127317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.128519 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.128526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:00.128759 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:00.129665 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:00.129683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:00.133598 (+ 3915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":357,"threads_started":1}
W20260809 22:38:00.134119 7153 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:00.122391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:49093 (local address 127.28.243.126:41471)
0809 22:38:00.122612 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:00.122620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:00.123063 (+ 443us) server_negotiation.cc:407] Connection header received
0809 22:38:00.123267 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:00.123274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:00.123372 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:00.123483 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:00.124523 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.125821 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.127488 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.128240 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.128399 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:00.133832 (+ 5433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:49093: BlockingRecv error: recv got EOF from 127.28.243.67:49093 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 22:38:00.134605 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:00.122787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:53149)
0809 22:38:00.123631 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:00.123649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:00.123800 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:00.124162 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:00.124169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:00.124871 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:00.125227 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:00.125243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.126824 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.126829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:00.128299 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:00.128321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.129433 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.129443 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:00.129662 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:00.130297 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:00.130319 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:00.134276 (+ 3957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":545,"threads_started":1}
W20260809 22:38:00.134991 7156 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:00.122818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:53149 (local address 127.28.243.126:41471)
0809 22:38:00.123418 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:00.123425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:00.123746 (+ 321us) server_negotiation.cc:407] Connection header received
0809 22:38:00.123926 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:00.123933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:00.124004 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:00.124118 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:00.125464 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.126668 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.128515 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.129215 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.129314 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:00.134692 (+ 5378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:53149: BlockingRecv error: recv got EOF from 127.28.243.65:53149 (error 108)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":313,"threads_started":1}
Aug 09 22:38:00 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:00 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:00.645937 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:00.636303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:35419)
0809 22:38:00.636948 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:00.636968 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:00.637088 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:00.637506 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:00.637512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:00.638035 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:00.638206 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:00.638215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.639387 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.639392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:00.640382 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:00.640395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.641293 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.641301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:00.641475 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:00.642298 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:00.642319 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:00.645646 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":302,"threads_started":1}
W20260809 22:38:00.646391 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:00.636683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:35419 (local address 127.28.243.126:41471)
0809 22:38:00.637149 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:00.637155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:00.637172 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:38:00.637225 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:00.637231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:00.637305 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:00.637395 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:00.638383 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.639192 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.640538 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:00.641106 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:00.641225 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:00.645912 (+ 4687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:35419: BlockingRecv error: recv got EOF from 127.28.243.66:35419 (error 108)
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":253,"threads_started":1}
Aug 09 22:38:01 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:01 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:01 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:01 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:01.149623 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:01.135653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:47497)
0809 22:38:01.136936 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:01.136957 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:01.137087 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:01.137651 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:01.137657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:01.138068 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:01.138247 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:01.138258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.139603 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.139610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:01.141373 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:01.141402 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.142776 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.142783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:01.142977 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:01.143762 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:01.143785 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:01.149255 (+ 5470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":1001,"threads_started":1}
I20260809 22:38:01.149935 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:01.135185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:56145)
0809 22:38:01.135823 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:01.135841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:01.136130 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:01.136532 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:01.136538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:01.137205 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:01.137469 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:01.137481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.139161 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.139165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:01.141150 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:01.141178 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.142677 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.142695 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:01.142913 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:01.143747 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:01.143774 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:01.149563 (+ 5789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":306,"threads_started":1}
W20260809 22:38:01.150024 7160 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:01.135298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:56145 (local address 127.28.243.126:41471)
0809 22:38:01.135579 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:01.135588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:01.135938 (+ 350us) server_negotiation.cc:407] Connection header received
0809 22:38:01.136287 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:01.136293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:01.136372 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:01.136502 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:01.137675 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.138890 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.141375 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.142312 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.142458 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:01.149753 (+ 7295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:56145: BlockingRecv error: recv got EOF from 127.28.243.67:56145 (error 108)
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":84}
W20260809 22:38:01.150336 7163 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:01.135885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:47497 (local address 127.28.243.126:41471)
0809 22:38:01.137346 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:01.137350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:01.137366 (+ 16us) server_negotiation.cc:407] Connection header received
0809 22:38:01.137405 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:01.137408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:01.137458 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:01.137544 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:01.138404 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.139351 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.141675 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.142472 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.142585 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:01.149514 (+ 6929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:47497: BlockingRecv error: recv got EOF from 127.28.243.65:47497 (error 108)
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":1315,"spinlock_wait_cycles":41216,"thread_start_us":1196,"threads_started":1}
Aug 09 22:38:01 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:01 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:01.657370 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:01.647091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:58053)
0809 22:38:01.647668 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:01.647691 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:01.647842 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:01.648282 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:01.648290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:01.648911 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:01.649157 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:01.649174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.650582 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.650590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:01.652080 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:01.652093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.652958 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.652962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:01.653158 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:01.653863 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:01.653883 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:01.656971 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":286,"threads_started":1}
W20260809 22:38:01.657578 7163 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:01.647166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:58053 (local address 127.28.243.126:41471)
0809 22:38:01.647415 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:01.647422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:01.647761 (+ 339us) server_negotiation.cc:407] Connection header received
0809 22:38:01.647981 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:01.647988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:01.648100 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:01.648205 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:01.649352 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.650404 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.652232 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:01.652770 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:01.652947 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:01.657312 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:58053: BlockingRecv error: recv got EOF from 127.28.243.66:58053 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 22:38:02 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:02 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:02 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:02 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:02.163184 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:02.150952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:48305)
0809 22:38:02.151817 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:02.151841 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:02.152082 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:02.152662 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:02.152666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:02.153301 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:02.153534 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:02.153548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.155403 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.155409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:02.157040 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:02.157058 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.158234 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.158244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:02.158418 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:02.159213 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:02.159244 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:02.162906 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":472,"threads_started":1}
I20260809 22:38:02.163558 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:02.151214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:60651)
0809 22:38:02.152125 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:02.152145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:02.152270 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:02.152896 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:02.152901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:02.153368 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:02.153579 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:02.153594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.155408 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.155414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:02.156998 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:02.157017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.158792 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.158800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:02.159025 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:02.159719 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:02.159736 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:02.163259 (+ 3523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":672,"threads_started":1}
W20260809 22:38:02.163942 7163 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:02.151073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:48305 (local address 127.28.243.126:41471)
0809 22:38:02.151399 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:02.151409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:02.151937 (+ 528us) server_negotiation.cc:407] Connection header received
0809 22:38:02.152264 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:02.152271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:02.152364 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:02.152452 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:02.153776 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.155200 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.157208 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.158051 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.158199 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:02.163193 (+ 4994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:48305: BlockingRecv error: recv got EOF from 127.28.243.65:48305 (error 108)
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":124}
W20260809 22:38:02.164111 7167 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:02.151285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:60651 (local address 127.28.243.126:41471)
0809 22:38:02.152530 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:02.152534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:02.152551 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:38:02.152673 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:02.152676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:02.152726 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:02.152811 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:02.153776 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.155200 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.157208 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.158336 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.158459 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:02.163737 (+ 5278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:60651: BlockingRecv error: recv got EOF from 127.28.243.67:60651 (error 108)
Metrics: {"mutex_wait_us":420,"server-negotiator.queue_time_us":1165,"thread_start_us":995,"threads_started":1}
Aug 09 22:38:02 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:02 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:02.668494 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:02.658793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:53133)
0809 22:38:02.659427 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:02.659444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:02.659569 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:02.659938 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:02.659942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:02.660443 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:02.660682 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:02.660692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.661793 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.661797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:02.662897 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:02.662910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.663662 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.663667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:02.663793 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:02.664670 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:02.664704 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:02.668212 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":331,"threads_started":1}
W20260809 22:38:02.668814 7167 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:02.658889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:53133 (local address 127.28.243.126:41471)
0809 22:38:02.659130 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:02.659140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:02.659536 (+ 396us) server_negotiation.cc:407] Connection header received
0809 22:38:02.659680 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:02.659683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:02.659748 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:02.659833 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:02.660836 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.661629 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.663045 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:02.663497 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:02.663625 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:02.668479 (+ 4854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:53133: BlockingRecv error: recv got EOF from 127.28.243.66:53133 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 22:38:03 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:03 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:03 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:03 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:03.175734 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:03.164680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:53015)
0809 22:38:03.165399 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:03.165438 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:03.165635 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:03.166446 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:03.166452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:03.166905 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:03.167097 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:03.167108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.168375 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.168381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:03.169828 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:03.169841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.170826 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.170833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:03.171001 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:03.171884 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:03.171902 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:03.175334 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":401,"threads_started":1}
W20260809 22:38:03.176036 7167 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:03.164828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:53015 (local address 127.28.243.126:41471)
0809 22:38:03.165012 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:03.165022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:03.165533 (+ 511us) server_negotiation.cc:407] Connection header received
0809 22:38:03.165763 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:03.165772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:03.165907 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:03.166008 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:03.167292 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.168188 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.170036 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.170628 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.170737 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:03.175619 (+ 4882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:53015: BlockingRecv error: recv got EOF from 127.28.243.65:53015 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":54}
I20260809 22:38:03.176131 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:03.165204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:34851)
0809 22:38:03.165840 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:03.165860 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:03.165998 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:03.166566 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:03.166569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:03.167001 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:03.167165 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:03.167174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.168613 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.168619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:03.170067 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:03.170082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.170994 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.170998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:03.171111 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:03.171827 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:03.171847 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:03.175726 (+ 3879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":269,"threads_started":1}
W20260809 22:38:03.176432 7171 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:03.165371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:34851 (local address 127.28.243.126:41471)
0809 22:38:03.166245 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:03.166251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:03.166269 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:38:03.166316 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:03.166325 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:03.166389 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:03.166502 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:03.167353 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.168380 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.170186 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.170810 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.170949 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:03.175939 (+ 4990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:34851: BlockingRecv error: recv got EOF from 127.28.243.67:34851 (error 108)
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":763,"thread_start_us":797,"threads_started":1}
Aug 09 22:38:03 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:03 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:03.680300 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:03.669578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:50401)
0809 22:38:03.670235 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:03.670251 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:03.670423 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:03.670807 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:03.670812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:03.671248 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:03.671406 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:03.671415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.673042 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.673047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:03.674510 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:03.674526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.675521 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.675527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:03.675678 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:03.676636 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:03.676665 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:03.680048 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":315,"threads_started":1}
W20260809 22:38:03.680567 7171 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:03.669774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:50401 (local address 127.28.243.126:41471)
0809 22:38:03.670039 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:03.670046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:03.670346 (+ 300us) server_negotiation.cc:407] Connection header received
0809 22:38:03.670584 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:03.670589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:03.670654 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:03.670766 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:03.671563 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.672753 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.674691 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:03.675231 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:03.675357 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:03.680269 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:50401: BlockingRecv error: recv got EOF from 127.28.243.66:50401 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 09 22:38:04 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:04 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:04 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:04 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:04.188501 7173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:04.176970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:41037)
0809 22:38:04.177719 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:04.177743 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:04.177944 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:04.178585 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:04.178589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:04.179108 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:04.179370 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:04.179386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.180946 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.180953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:04.182457 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:04.182472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.183486 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.183491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:04.183741 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:04.184636 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:04.184666 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:04.188147 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":464,"threads_started":1}
W20260809 22:38:04.188832 7171 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:04.177060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:41037 (local address 127.28.243.126:41471)
0809 22:38:04.177309 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:04.177318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:04.177822 (+ 504us) server_negotiation.cc:407] Connection header received
0809 22:38:04.178118 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:04.178126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:04.178226 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:04.178313 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:04.179560 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.180761 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.182613 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.183273 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.183404 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:04.188369 (+ 4965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:41037: BlockingRecv error: recv got EOF from 127.28.243.65:41037 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 22:38:04.188957 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:04.177252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:39275)
0809 22:38:04.178644 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:04.178665 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:04.178800 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:04.179443 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:04.179451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:04.180002 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:04.180203 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:04.180214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.182140 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.182144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:04.183475 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:04.183491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.184702 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.184708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:04.184826 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:04.185441 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:04.185460 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:04.188645 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":1139,"threads_started":1}
W20260809 22:38:04.189368 7175 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:04.177463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:39275 (local address 127.28.243.126:41471)
0809 22:38:04.178248 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:04.178256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:04.178773 (+ 517us) server_negotiation.cc:407] Connection header received
0809 22:38:04.178942 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:04.178948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:04.179039 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:04.179169 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:04.180418 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.181920 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.183675 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.184397 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.184610 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:04.189054 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:39275: BlockingRecv error: recv got EOF from 127.28.243.67:39275 (error 108)
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":525,"threads_started":1}
Aug 09 22:38:04 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:04 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:04.696213 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:04.681323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:33959)
0809 22:38:04.681877 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:04.681894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:04.682081 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:04.682467 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:04.682471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:04.683039 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:04.683251 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:04.683264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.684739 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.684744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:04.686020 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:04.686035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.686902 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.686907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:04.687034 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:04.692024 (+ 4990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:04.692122 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:04.695825 (+ 3703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":311,"threads_started":1}
W20260809 22:38:04.696334 7175 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:04.681412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:33959 (local address 127.28.243.126:41471)
0809 22:38:04.681618 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:04.681627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:04.681995 (+ 368us) server_negotiation.cc:407] Connection header received
0809 22:38:04.682219 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:04.682226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:04.682293 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:04.682416 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:04.683468 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.684559 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.686198 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:04.686733 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:04.686861 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:04.696041 (+ 9180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:33959: BlockingRecv error: recv got EOF from 127.28.243.66:33959 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 22:38:05 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:05 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:05 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:05 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:05.201095 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:05.190013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:56843)
0809 22:38:05.190866 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:05.190884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:05.191004 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:05.191600 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:05.191603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:05.191988 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:05.192227 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:05.192242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.193736 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.193740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:05.194864 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:05.194879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.195933 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.195937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:05.196083 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:05.196792 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:05.196820 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:05.200724 (+ 3904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":525,"threads_started":1}
W20260809 22:38:05.201325 7179 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:05.190297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:56843 (local address 127.28.243.126:41471)
0809 22:38:05.191257 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:05.191263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:05.191281 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:38:05.191334 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:05.191338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:05.191428 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:05.191549 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:05.192380 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.193554 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.195092 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.195766 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.195914 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:05.200927 (+ 5013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:56843: BlockingRecv error: recv got EOF from 127.28.243.67:56843 (error 108)
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":1007,"threads_started":1}
I20260809 22:38:05.201491 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:05.189807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:57477)
0809 22:38:05.190545 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:05.190574 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:05.190762 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:05.191276 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:05.191281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:05.191808 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:05.191979 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:05.191988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.193273 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.193281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:05.194582 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:05.194596 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.195581 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.195590 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:05.195794 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:05.196645 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:05.196697 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:05.201147 (+ 4450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":424,"threads_started":1}
W20260809 22:38:05.201676 7175 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:05.189959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:57477 (local address 127.28.243.126:41471)
0809 22:38:05.190182 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:05.190192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:05.190677 (+ 485us) server_negotiation.cc:407] Connection header received
0809 22:38:05.190982 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:05.190988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:05.191103 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:05.191206 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:05.192140 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.193090 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.194757 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.195354 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.195454 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:05.201394 (+ 5940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:57477: BlockingRecv error: recv got EOF from 127.28.243.65:57477 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 09 22:38:05 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:05 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:05.707269 7180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:05.697524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:50293)
0809 22:38:05.698216 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:05.698234 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:05.698360 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:05.698756 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:05.698761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:05.699234 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:05.699465 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:05.699475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.700527 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.700531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:05.701690 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:05.701704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.702666 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.702675 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:05.702843 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:05.703524 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:05.703557 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:05.707000 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":437,"threads_started":1}
W20260809 22:38:05.707471 7175 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:05.697616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:50293 (local address 127.28.243.126:41471)
0809 22:38:05.697785 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:05.697790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:05.698313 (+ 523us) server_negotiation.cc:407] Connection header received
0809 22:38:05.698473 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:05.698477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:05.698555 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:05.698640 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:05.699616 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.700384 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.701856 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:05.702459 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:05.702607 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:05.707215 (+ 4608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:50293: BlockingRecv error: recv got EOF from 127.28.243.66:50293 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 22:38:06 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:06 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:06 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:06 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:06.218062 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:06.202452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:55401)
0809 22:38:06.205223 (+ 2771us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:06.205254 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:06.205519 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:06.206239 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:06.206247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:06.206844 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:06.207081 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:06.207095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.208917 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.208925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:06.211059 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:06.211078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.212526 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.212535 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:06.212842 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:06.213741 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:06.213774 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:06.217693 (+ 3919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2570,"thread_start_us":2441,"threads_started":1}
W20260809 22:38:06.218196 7175 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:06.202648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:55401 (local address 127.28.243.126:41471)
0809 22:38:06.202913 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:06.202921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:06.205379 (+ 2458us) server_negotiation.cc:407] Connection header received
0809 22:38:06.205710 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:06.205716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:06.205802 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:06.205886 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:06.207370 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.208683 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.211227 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.212156 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.212277 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:06.217918 (+ 5641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:55401: BlockingRecv error: recv got EOF from 127.28.243.67:55401 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 22:38:06.218327 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:06.202705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:59809)
0809 22:38:06.205825 (+ 3120us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:06.205848 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:06.205982 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:06.206489 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:06.206496 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:06.207044 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:06.207425 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:06.207451 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.208915 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.208921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:06.210629 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:06.210649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.212205 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.212213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:06.212436 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:06.213796 (+ 1360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:06.213823 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:06.217953 (+ 4130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2993,"thread_start_us":2847,"threads_started":1}
W20260809 22:38:06.218547 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:06.202733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:59809 (local address 127.28.243.126:41471)
0809 22:38:06.205637 (+ 2904us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:06.205643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:06.205972 (+ 329us) server_negotiation.cc:407] Connection header received
0809 22:38:06.206211 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:06.206216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:06.206287 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:06.206381 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:06.207629 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.208751 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.210831 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.211851 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.212103 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:06.218183 (+ 6080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:59809: BlockingRecv error: recv got EOF from 127.28.243.65:59809 (error 108)
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":2759,"thread_start_us":2550,"threads_started":1}
Aug 09 22:38:06 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:06 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:06.717455 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:06.708261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:40069)
0809 22:38:06.708910 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:06.708929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:06.709060 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:06.709420 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:06.709424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:06.709878 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:06.710086 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:06.710096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.711284 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.711289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:06.712428 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:06.712443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.713188 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.713193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:06.713319 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:06.713980 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:06.714000 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:06.717194 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":388,"threads_started":1}
W20260809 22:38:06.717692 7183 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:06.708516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:40069 (local address 127.28.243.126:41471)
0809 22:38:06.708789 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:06.708796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:06.709064 (+ 268us) server_negotiation.cc:407] Connection header received
0809 22:38:06.709154 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:06.709158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:06.709225 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:06.709312 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:06.710261 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.711146 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.712611 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:06.713042 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:06.713144 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:06.717441 (+ 4297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:40069: BlockingRecv error: recv got EOF from 127.28.243.66:40069 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 09 22:38:07 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:07 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:07.230911 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:07.219522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:39555)
0809 22:38:07.220717 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:07.220737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:07.220912 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:07.221301 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:07.221306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:07.221753 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:07.221912 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:07.221921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.223065 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.223071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:07.224021 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:07.224033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.225124 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.225129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:07.225345 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:07.226228 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:07.226255 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:07.230653 (+ 4398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1085,"thread_start_us":950,"threads_started":1}
W20260809 22:38:07.231249 7186 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:07.219564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:39555 (local address 127.28.243.126:41471)
0809 22:38:07.220294 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:07.220304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:07.220838 (+ 534us) server_negotiation.cc:407] Connection header received
0809 22:38:07.221057 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:07.221062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:07.221142 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:07.221260 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:07.222077 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.222912 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.224172 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.224760 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.224895 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:07.230933 (+ 6038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:39555: BlockingRecv error: recv got EOF from 127.28.243.65:39555 (error 108)
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":419,"threads_started":1}
Aug 09 22:38:07 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:07 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:07.237996 7187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:07.219782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:45149)
0809 22:38:07.225006 (+ 5224us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:07.225030 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:07.225156 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:07.229248 (+ 4092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:07.229257 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:07.229936 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:07.230229 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:07.230245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.231731 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.231736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:07.232793 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:07.232809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.233509 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.233514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:07.233675 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:07.234412 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:07.234432 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:07.237573 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5096,"thread_start_us":4974,"threads_started":1}
W20260809 22:38:07.238160 7188 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:07.220163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:45149 (local address 127.28.243.126:41471)
0809 22:38:07.228667 (+ 8504us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:07.228674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:07.228698 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:38:07.228763 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:07.228768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:07.228881 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:07.229147 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:07.230380 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.231571 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.232930 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.233354 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.233448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:07.237840 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:45149: BlockingRecv error: recv got EOF from 127.28.243.67:45149 (error 108)
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":8305,"thread_start_us":8176,"threads_started":1}
Aug 09 22:38:07 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:07 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:07.728824 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:07.719214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:44585)
0809 22:38:07.719846 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:07.719863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:07.720042 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:07.720411 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:07.720418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:07.720890 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:07.721112 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:07.721123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.722329 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.722336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:07.723316 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:07.723329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.724090 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.724096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:07.724293 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:07.725206 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:07.725226 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:07.728416 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":399,"threads_started":1}
W20260809 22:38:07.728993 7188 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:07.719331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:44585 (local address 127.28.243.126:41471)
0809 22:38:07.719577 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:07.719584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:07.719957 (+ 373us) server_negotiation.cc:407] Connection header received
0809 22:38:07.720174 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:07.720180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:07.720260 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:07.720365 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:07.721259 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.722163 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.723474 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:07.723908 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:07.724047 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:07.728751 (+ 4704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:44585: BlockingRecv error: recv got EOF from 127.28.243.66:44585 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 22:38:08 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:08 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:08.243930 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:08.231968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:34847)
0809 22:38:08.232689 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:08.232709 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:08.232840 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:08.233332 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:08.233336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:08.233811 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:08.233977 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:08.233986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.235214 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.235218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:08.236269 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:08.236283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.237508 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.237513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:08.237808 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:08.238831 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:08.238859 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:08.243561 (+ 4702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":384,"threads_started":1}
W20260809 22:38:08.244103 7191 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:08.232099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:34847 (local address 127.28.243.126:41471)
0809 22:38:08.232672 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:08.232677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:08.232820 (+ 143us) server_negotiation.cc:407] Connection header received
0809 22:38:08.233066 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:08.233071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:08.233144 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:08.233255 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:08.234202 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.235057 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.236502 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.237274 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.237424 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:08.243786 (+ 6362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:34847: BlockingRecv error: recv got EOF from 127.28.243.65:34847 (error 108)
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":321,"threads_started":1}
Aug 09 22:38:08 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:08 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:08.251374 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:08.239078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:57959)
0809 22:38:08.242081 (+ 3003us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:08.242104 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:08.242286 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:08.242782 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:08.242785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:08.243240 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:08.243412 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:08.243422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.245096 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.245107 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:08.246136 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:08.246150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.247025 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.247033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:08.247250 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:08.247936 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:08.247956 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:08.251108 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2856,"thread_start_us":2744,"threads_started":1}
W20260809 22:38:08.251940 7193 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:08.239134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:57959 (local address 127.28.243.126:41471)
0809 22:38:08.241904 (+ 2770us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:08.241910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:08.242232 (+ 322us) server_negotiation.cc:407] Connection header received
0809 22:38:08.242512 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:08.242520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:08.242616 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:08.242744 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:08.243581 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.244908 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.246313 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.246844 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.246993 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:08.251415 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:57959: BlockingRecv error: recv got EOF from 127.28.243.67:57959 (error 108)
Metrics: {"server-negotiator.queue_time_us":2597,"thread_start_us":2413,"threads_started":1}
Aug 09 22:38:08 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:08 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:08.739516 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:08.730126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:36539)
0809 22:38:08.730812 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:08.730831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:08.730959 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:08.731268 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:08.731281 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:08.731749 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:08.731926 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:08.731936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.733162 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.733167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:08.734149 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:08.734161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.735010 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.735016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:08.735165 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:08.735872 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:08.735892 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:08.739205 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":381,"threads_started":1}
W20260809 22:38:08.739761 7193 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:08.730230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:36539 (local address 127.28.243.126:41471)
0809 22:38:08.730415 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:08.730419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:08.730915 (+ 496us) server_negotiation.cc:407] Connection header received
0809 22:38:08.731064 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:08.731067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:08.731132 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:08.731218 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:08.732073 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.732986 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.734302 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:08.734740 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:08.734839 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:08.739460 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:36539: BlockingRecv error: recv got EOF from 127.28.243.66:36539 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 22:38:09 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:09 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:09.254534 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:09.245107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:37649)
0809 22:38:09.245632 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:09.245650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:09.245764 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:09.246160 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:09.246163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:09.246535 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:09.246750 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:09.246759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.247814 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.247819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:09.248838 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:09.248851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.249634 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.249638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:09.249759 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:09.250562 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:09.250583 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:09.254206 (+ 3623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":287,"threads_started":1}
W20260809 22:38:09.254772 7196 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:09.245224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:37649 (local address 127.28.243.126:41471)
0809 22:38:09.245686 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:09.245691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:09.245708 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:38:09.245875 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:09.245878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:09.245940 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:09.246099 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:09.246892 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.247662 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.249025 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.249481 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.249580 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:09.254458 (+ 4878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:37649: BlockingRecv error: recv got EOF from 127.28.243.65:37649 (error 108)
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
Aug 09 22:38:09 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:09 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:09.262513 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:09.252518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:57161)
0809 22:38:09.253169 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:09.253196 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:09.253340 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:09.253993 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:09.253999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:09.254591 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:09.254844 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:09.254864 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.256047 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.256051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:09.257310 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:09.257324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.258310 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.258315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:09.258443 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:09.259115 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:09.259132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:09.262247 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
W20260809 22:38:09.262873 7199 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:09.252614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:57161 (local address 127.28.243.126:41471)
0809 22:38:09.253426 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:09.253432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:09.253452 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:38:09.253506 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:09.253511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:09.253587 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:09.253708 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:09.255038 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.255815 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.257485 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.258095 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.258216 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:09.262507 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:57161: BlockingRecv error: recv got EOF from 127.28.243.67:57161 (error 108)
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":460,"threads_started":1}
Aug 09 22:38:09 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:09 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:09.756314 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:09.740714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:50767)
0809 22:38:09.741519 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:09.741535 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:09.741713 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:09.742219 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:09.742223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:09.742744 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:09.743023 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:09.743036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.744171 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.744176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:09.745318 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:09.745334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.746214 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.746220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:09.746356 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:09.751764 (+ 5408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:09.751787 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:09.755874 (+ 4087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":551,"threads_started":1}
W20260809 22:38:09.756422 7199 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:09.740919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:50767 (local address 127.28.243.126:41471)
0809 22:38:09.741122 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:09.741128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:09.741623 (+ 495us) server_negotiation.cc:407] Connection header received
0809 22:38:09.741896 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:09.741901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:09.742028 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:09.742157 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:09.743187 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.743995 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.745497 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:09.746040 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:09.746159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:09.756141 (+ 9982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:50767: BlockingRecv error: recv got EOF from 127.28.243.66:50767 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 09 22:38:10 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:10 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.65@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:10.266409 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:10.255886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.65:35901)
0809 22:38:10.256653 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:10.256671 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:10.256827 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:10.257296 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:10.257301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:10.257745 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:10.257918 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:10.257927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.258934 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.258939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:10.259973 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:10.259987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.261102 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.261107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:10.261281 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:10.262074 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:10.262093 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:10.266016 (+ 3923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":408,"threads_started":1}
W20260809 22:38:10.266459 7199 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:10.256155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:35901 (local address 127.28.243.126:41471)
0809 22:38:10.256439 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:10.256446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:10.256756 (+ 310us) server_negotiation.cc:407] Connection header received
0809 22:38:10.257024 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:10.257031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:10.257117 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:10.257223 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:10.258064 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.258788 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.260183 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.260891 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.261057 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:10.266237 (+ 5180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.65:35901: BlockingRecv error: recv got EOF from 127.28.243.65:35901 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 09 22:38:10 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:10 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.67@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:10.274663 7202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:10.263527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.67:53377)
0809 22:38:10.265221 (+ 1694us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:10.265246 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:10.265394 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:10.265788 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:10.265793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:10.266266 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:10.266524 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:10.266539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.267745 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.267749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:10.268786 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:10.268799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.269539 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.269545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:10.269704 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:10.270350 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:10.270369 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:10.274229 (+ 3860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":1441,"threads_started":1}
W20260809 22:38:10.274880 7203 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:10.263631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:53377 (local address 127.28.243.126:41471)
0809 22:38:10.265107 (+ 1476us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:10.265113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:10.265360 (+ 247us) server_negotiation.cc:407] Connection header received
0809 22:38:10.265551 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:10.265556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:10.265616 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:10.265726 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:10.266735 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.267593 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.268919 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.269373 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.269474 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:10.274491 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.67:53377: BlockingRecv error: recv got EOF from 127.28.243.67:53377 (error 108)
Metrics: {"server-negotiator.queue_time_us":1310,"thread_start_us":1110,"threads_started":1}
Aug 09 22:38:10 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 22:38:10 dist-test-slave-rgb1 krb5kdc[6030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.243.66@KRBTEST.COM for oryx/127.28.243.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 22:38:10.768397 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:10.757727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41471 (local address 127.28.243.66:41341)
0809 22:38:10.758470 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:10.758496 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:10.758685 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:10.759100 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:10.759105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:10.759778 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:10.759986 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:10.759998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.761312 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.761318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:10.762645 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:10.762660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.763552 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.763559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:10.763748 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:10.764400 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:10.764421 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:10.768102 (+ 3681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.243.126:41471: Server oryx/127.28.243.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":420,"threads_started":1}
W20260809 22:38:10.768807 7203 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:10.757916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:41341 (local address 127.28.243.126:41471)
0809 22:38:10.758231 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:10.758241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:10.758607 (+ 366us) server_negotiation.cc:407] Connection header received
0809 22:38:10.758843 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:10.758849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:10.758931 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:10.759056 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:10.760182 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.761122 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.762820 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:10.763363 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:10.763535 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:10.768336 (+ 4801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.243.66:41341: BlockingRecv error: recv got EOF from 127.28.243.66:41341 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260809 22:38:11.044021 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6672
I20260809 22:38:11.059005 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:11.045483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.28.243.65:41157)
0809 22:38:11.046148 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:11.046157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:11.046187 (+ 30us) server_negotiation.cc:407] Connection header received
0809 22:38:11.046267 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:11.046274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:11.046521 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:11.046714 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:11.048508 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.049703 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.051103 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.051715 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.051846 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:11.055110 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:11.055120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:11.055136 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:11.055184 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:11.058384 (+ 3200us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 22:38:11.058714 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41868: Request ticket server kudu/127.28.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":333,"threads_started":1}
W20260809 22:38:11.059103 7207 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41868: Request ticket server kudu/127.28.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 22:38:11.059468 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6672: Not authorized: Client connection negotiation failed: client connection to 127.28.243.65:41157: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 22:38:11.059736 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6672
I20260809 22:38:11.081362 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6837
I20260809 22:38:11.095685 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:11.082729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.28.243.66:39003)
0809 22:38:11.083508 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:11.083517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:11.083552 (+ 35us) server_negotiation.cc:407] Connection header received
0809 22:38:11.083627 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:11.083634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:11.083880 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:11.084051 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:11.085582 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.086902 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.088358 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.089122 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.089257 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:11.092166 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:11.092175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:11.092191 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:11.092232 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:11.095132 (+ 2900us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 22:38:11.095436 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48536: Request ticket server kudu/127.28.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":421,"threads_started":1}
W20260809 22:38:11.095781 7208 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48536: Request ticket server kudu/127.28.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 22:38:11.096048 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6837: Not authorized: Client connection negotiation failed: client connection to 127.28.243.66:39003: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 22:38:11.096190 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6837
I20260809 22:38:11.116709 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6985
I20260809 22:38:11.135473 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:11.118309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.28.243.67:34917)
0809 22:38:11.119386 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:11.119395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:11.119506 (+ 111us) server_negotiation.cc:407] Connection header received
0809 22:38:11.119627 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:11.119636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:11.119893 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:11.120082 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:11.122497 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.124124 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.125818 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.126494 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.126639 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:11.131373 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:11.131386 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:11.131403 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:11.131463 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:11.134854 (+ 3391us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 22:38:11.135206 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49120: Request ticket server kudu/127.28.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":711,"threads_started":1}
W20260809 22:38:11.135574 7209 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49120: Request ticket server kudu/127.28.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 22:38:11.135919 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6985: Not authorized: Client connection negotiation failed: client connection to 127.28.243.67:34917: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.28.243.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 22:38:11.136162 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6985
I20260809 22:38:11.155625 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 6597
I20260809 22:38:11.157361 6664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:11.290720 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 6597
2026-08-09T22:38:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21930 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T22:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:38:11Z Disabled control of system clock
I20260809 22:38:11.389197 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36645
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:36645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 22:38:11.756064 7214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:11.756390 7214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:11.756536 7214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:11.764199 7214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:38:11.764330 7214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:11.764407 7214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:11.764505 7214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:38:11.764631 7214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:38:11.774870 7214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:36645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:11.776623 7214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:11.778697 7214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:11.793191 7219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:11.793260 7220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:11.793291 7224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:11.794346 7214 server_base.cc:1034] running on GCE node
I20260809 22:38:11.795214 7214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:11.796819 7214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:11.798084 7214 hybrid_clock.cc:648] HybridClock initialized: now 1786315091798030 us; error 44 us; skew 500 ppm
I20260809 22:38:11.798509 7214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:11.801293 7214 webserver.cc:460] Webserver started at http://127.28.243.126:43123/ using document root <none> and password file <none>
I20260809 22:38:11.802099 7214 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:11.802196 7214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:11.802574 7214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:11.805791 7214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "aee201b7eb9a4dfba5f58c76628f5f5d"
format_stamp: "Formatted at 2026-08-09 22:38:11 on dist-test-slave-rgb1"
server_key: "9b1621af478c0659891da5ca9d1e560b"
server_key_iv: "cf8a440d77623c26346a2c494db7f24a"
server_key_version: "encryptionkey@0"
I20260809 22:38:11.806646 7214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "aee201b7eb9a4dfba5f58c76628f5f5d"
format_stamp: "Formatted at 2026-08-09 22:38:11 on dist-test-slave-rgb1"
server_key: "9b1621af478c0659891da5ca9d1e560b"
server_key_iv: "cf8a440d77623c26346a2c494db7f24a"
server_key_version: "encryptionkey@0"
I20260809 22:38:11.812489 7214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 22:38:11.816677 7233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:11.818334 7214 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:38:11.818547 7214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "aee201b7eb9a4dfba5f58c76628f5f5d"
format_stamp: "Formatted at 2026-08-09 22:38:11 on dist-test-slave-rgb1"
server_key: "9b1621af478c0659891da5ca9d1e560b"
server_key_iv: "cf8a440d77623c26346a2c494db7f24a"
server_key_version: "encryptionkey@0"
I20260809 22:38:11.818830 7214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:11.850034 7214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:11.851038 7214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:11.851392 7214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:11.874419 7214 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:36645
I20260809 22:38:11.874454 7287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:36645 every 8 connection(s)
I20260809 22:38:11.876238 7214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:38:11.880400 7288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:38:11.883991 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 7214
I20260809 22:38:11.884315 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:38:11.885145 29645 external_mini_cluster.cc:1428] Setting key b13c0b856da62c73a3378fe0b7347c21
I20260809 22:38:11.892891 7288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d: Bootstrap starting.
I20260809 22:38:11.896816 7288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d: Neither blocks nor log segments found. Creating new log.
I20260809 22:38:11.898171 7288 log.cc:824] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:38:11.902463 7288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d: No bootstrap required, opened a new log
I20260809 22:38:11.902582 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:11.887645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.28.243.126:36645)
0809 22:38:11.889499 (+ 1854us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:11.889515 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:11.889569 (+ 54us) server_negotiation.cc:407] Connection header received
0809 22:38:11.891506 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:11.891531 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:11.892037 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:11.892415 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:11.893234 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.894619 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.896536 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:11.897571 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:11.897767 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:11.899329 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:11.899348 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:11.899372 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:11.900353 (+ 981us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:11.900440 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:11.900464 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:11.900644 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:11.900913 (+ 269us) server_negotiation.cc:299] Negotiation successful
0809 22:38:11.901445 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":504,"threads_started":1}
I20260809 22:38:11.911943 7288 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36645 } }
I20260809 22:38:11.912417 7288 raft_consensus.cc:374] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:38:11.912542 7288 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee201b7eb9a4dfba5f58c76628f5f5d, State: Initialized, Role: FOLLOWER
I20260809 22:38:11.913300 7288 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36645 } }
I20260809 22:38:11.913604 7288 raft_consensus.cc:388] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:38:11.913779 7288 raft_consensus.cc:482] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:38:11.913991 7288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:38:11.916508 7288 raft_consensus.cc:504] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36645 } }
I20260809 22:38:11.917215 7288 leader_election.cc:302] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aee201b7eb9a4dfba5f58c76628f5f5d; no voters:
I20260809 22:38:11.918119 7288 leader_election.cc:288] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:38:11.918386 7293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:38:11.919381 7293 raft_consensus.cc:686] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [term 1 LEADER]: Becoming Leader. State: Replica: aee201b7eb9a4dfba5f58c76628f5f5d, State: Running, Role: LEADER
I20260809 22:38:11.920068 7293 consensus_queue.cc:237] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36645 } }
I20260809 22:38:11.920687 7288 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:38:11.924044 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aee201b7eb9a4dfba5f58c76628f5f5d. Latest consensus state: current_term: 1 leader_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36645 } } }
I20260809 22:38:11.924109 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aee201b7eb9a4dfba5f58c76628f5f5d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36645 } } }
I20260809 22:38:11.924839 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [sys.catalog]: This master's current role is: LEADER
I20260809 22:38:11.924830 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d [sys.catalog]: This master's current role is: LEADER
I20260809 22:38:11.929728 7305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:38:11.935902 7305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:38:11.948236 7305 catalog_manager.cc:1347] Generated new cluster ID: cf3d983f24514cb3b3730cc03c24cc2d
I20260809 22:38:11.948496 7305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:38:11.985749 7305 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:38:11.988911 7305 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:38:12.007508 7305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aee201b7eb9a4dfba5f58c76628f5f5d: Generated new TSK 0
I20260809 22:38:12.008888 7305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:38:12.023351 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36645
--builtin_ntp_servers=127.28.243.84:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 22:38:12.431308 7316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:12.431682 7316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:12.431854 7316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:12.442118 7316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:12.442844 7316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:12.443171 7316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:38:12.454039 7316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:12.455467 7316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:12.457648 7316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:12.475147 7321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:12.477432 7322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:12.477452 7324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:12.478655 7316 server_base.cc:1034] running on GCE node
I20260809 22:38:12.479463 7316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:12.480726 7316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:12.482013 7316 hybrid_clock.cc:648] HybridClock initialized: now 1786315092481966 us; error 55 us; skew 500 ppm
I20260809 22:38:12.482478 7316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:12.485690 7316 webserver.cc:460] Webserver started at http://127.28.243.65:34003/ using document root <none> and password file <none>
I20260809 22:38:12.486606 7316 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:12.486770 7316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:12.487150 7316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:12.490860 7316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "74fa693e70934eafa98724bd0a550f76"
format_stamp: "Formatted at 2026-08-09 22:38:12 on dist-test-slave-rgb1"
server_key: "dddbaea0032fa5d9fc468594579b5f10"
server_key_iv: "5c66e36137e914085b352fba4fc9452b"
server_key_version: "encryptionkey@0"
I20260809 22:38:12.492470 7316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "74fa693e70934eafa98724bd0a550f76"
format_stamp: "Formatted at 2026-08-09 22:38:12 on dist-test-slave-rgb1"
server_key: "dddbaea0032fa5d9fc468594579b5f10"
server_key_iv: "5c66e36137e914085b352fba4fc9452b"
server_key_version: "encryptionkey@0"
I20260809 22:38:12.499229 7316 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260809 22:38:12.503669 7331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:12.505668 7316 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 22:38:12.506160 7316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "74fa693e70934eafa98724bd0a550f76"
format_stamp: "Formatted at 2026-08-09 22:38:12 on dist-test-slave-rgb1"
server_key: "dddbaea0032fa5d9fc468594579b5f10"
server_key_iv: "5c66e36137e914085b352fba4fc9452b"
server_key_version: "encryptionkey@0"
I20260809 22:38:12.506508 7316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:12.536365 7316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:12.537379 7316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:12.537729 7316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:12.538833 7316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:12.540475 7316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:12.540633 7316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:12.540771 7316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:12.540863 7316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:12.580443 7316 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:45271
I20260809 22:38:12.580549 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:45271 every 8 connection(s)
I20260809 22:38:12.581871 7316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:38:12.592813 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 7316
I20260809 22:38:12.593046 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:38:12.594094 29645 external_mini_cluster.cc:1428] Setting key f7f1848a29058ff3d66cafbe7db1753a
I20260809 22:38:12.597734 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:12.584765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:33615 (local address 127.28.243.126:36645)
0809 22:38:12.585297 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:12.585303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:12.586409 (+ 1106us) server_negotiation.cc:407] Connection header received
0809 22:38:12.588339 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:12.588345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:12.588447 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:12.588608 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:12.590278 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:12.591202 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:12.593597 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:12.594341 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:12.594693 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:12.596666 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:12.596672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:12.596675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:12.596696 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:12.596738 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:12.596751 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:12.596862 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:12.597228 (+ 366us) server_negotiation.cc:299] Negotiation successful
0809 22:38:12.597343 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":286,"threads_started":1}
I20260809 22:38:12.598031 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36645
--builtin_ntp_servers=127.28.243.84:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 22:38:12.598902 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:12.585257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36645 (local address 127.28.243.65:33615)
0809 22:38:12.586232 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:12.586283 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:12.588028 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:12.588872 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:12.588881 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:12.588983 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:12.590064 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:12.590094 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:12.591470 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:12.591475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:12.593346 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:12.593364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:12.594824 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:12.594830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:12.595269 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:12.596156 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:38:12.596160 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:12.596219 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:38:12.596343 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:12.596889 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:12.596905 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:12.596913 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 22:38:12.597147 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 22:38:12.597624 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":294,"threads_started":1}
I20260809 22:38:12.600868 7451 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36645
I20260809 22:38:12.601217 7451 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:12.602039 7451 heartbeater.cc:511] Master 127.28.243.126:36645 requested a full tablet report, sending...
I20260809 22:38:12.604521 7251 ts_manager.cc:194] Registered new tserver with Master: 74fa693e70934eafa98724bd0a550f76 (127.28.243.65:45271)
I20260809 22:38:12.607290 7251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.65:33615
W20260809 22:38:12.964226 7455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:12.964613 7455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:12.964771 7455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:12.974138 7455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:12.974284 7455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:12.974493 7455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:38:12.984504 7455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:12.985910 7455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:12.987782 7455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:13.003271 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:13.003232 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:13.003358 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:13.004320 7455 server_base.cc:1034] running on GCE node
I20260809 22:38:13.005103 7455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:13.006214 7455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:13.007441 7455 hybrid_clock.cc:648] HybridClock initialized: now 1786315093007376 us; error 67 us; skew 500 ppm
I20260809 22:38:13.007812 7455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:13.010457 7455 webserver.cc:460] Webserver started at http://127.28.243.66:46077/ using document root <none> and password file <none>
I20260809 22:38:13.011240 7455 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:13.011335 7455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:13.011792 7455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:13.014941 7455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "0d71ada74d7b450da5754c02da4fe2a6"
format_stamp: "Formatted at 2026-08-09 22:38:13 on dist-test-slave-rgb1"
server_key: "0f700cc9cd7d73c235b11a333b891a0a"
server_key_iv: "ab2d3813cdf6a2a4cf83296c4c6f26cc"
server_key_version: "encryptionkey@0"
I20260809 22:38:13.015861 7455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "0d71ada74d7b450da5754c02da4fe2a6"
format_stamp: "Formatted at 2026-08-09 22:38:13 on dist-test-slave-rgb1"
server_key: "0f700cc9cd7d73c235b11a333b891a0a"
server_key_iv: "ab2d3813cdf6a2a4cf83296c4c6f26cc"
server_key_version: "encryptionkey@0"
I20260809 22:38:13.022749 7455 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260809 22:38:13.026703 7470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:13.028416 7455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:38:13.028687 7455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "0d71ada74d7b450da5754c02da4fe2a6"
format_stamp: "Formatted at 2026-08-09 22:38:13 on dist-test-slave-rgb1"
server_key: "0f700cc9cd7d73c235b11a333b891a0a"
server_key_iv: "ab2d3813cdf6a2a4cf83296c4c6f26cc"
server_key_version: "encryptionkey@0"
I20260809 22:38:13.028931 7455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:13.052814 7455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:13.053753 7455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:13.054106 7455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:13.055218 7455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:13.056907 7455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:13.057004 7455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:13.057152 7455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:13.057255 7455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:13.096668 7455 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:43817
I20260809 22:38:13.096858 7586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:43817 every 8 connection(s)
I20260809 22:38:13.098055 7455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:38:13.102747 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 7455
I20260809 22:38:13.102950 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:38:13.103452 29645 external_mini_cluster.cc:1428] Setting key 255a26e3e75759e81f9b301911a33020
I20260809 22:38:13.106925 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36645
--builtin_ntp_servers=127.28.243.84:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 22:38:13.113569 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:13.101165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:59771 (local address 127.28.243.126:36645)
0809 22:38:13.102365 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:13.102372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:13.102932 (+ 560us) server_negotiation.cc:407] Connection header received
0809 22:38:13.105284 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:13.105289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:13.105400 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:13.105516 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:13.107116 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.108009 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.109973 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.110469 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.110635 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:13.112551 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:13.112599 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:13.112605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:13.112632 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:13.112686 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:13.112701 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:13.112805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:13.113073 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 22:38:13.113194 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":711,"threads_started":1}
I20260809 22:38:13.114686 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:13.101623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36645 (local address 127.28.243.66:59771)
0809 22:38:13.102750 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:13.102798 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:13.104939 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:13.105947 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:13.105957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:13.106076 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:13.106903 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:13.106927 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.108221 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.108226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:13.109771 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:13.109786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.110759 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.110768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:13.111258 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:13.112125 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:38:13.112131 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:13.112205 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:38:13.112305 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:13.112806 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:13.112820 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:13.112832 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 22:38:13.113059 (+ 227us) client_negotiation.cc:240] Negotiation successful
0809 22:38:13.113528 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":372,"threads_started":1}
I20260809 22:38:13.116748 7587 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36645
I20260809 22:38:13.117261 7587 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:13.118284 7587 heartbeater.cc:511] Master 127.28.243.126:36645 requested a full tablet report, sending...
I20260809 22:38:13.120765 7251 ts_manager.cc:194] Registered new tserver with Master: 0d71ada74d7b450da5754c02da4fe2a6 (127.28.243.66:43817)
I20260809 22:38:13.122115 7251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.66:59771
W20260809 22:38:13.487109 7592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:13.487419 7592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:13.487576 7592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:13.495966 7592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:13.496099 7592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:13.496284 7592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:38:13.506179 7592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:13.507558 7592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:13.509498 7592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:13.524500 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:13.524610 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:13.524610 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:13.526268 7592 server_base.cc:1034] running on GCE node
I20260809 22:38:13.526963 7592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:13.528112 7592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:13.529382 7592 hybrid_clock.cc:648] HybridClock initialized: now 1786315093529321 us; error 50 us; skew 500 ppm
I20260809 22:38:13.529793 7592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:13.532855 7592 webserver.cc:460] Webserver started at http://127.28.243.67:42029/ using document root <none> and password file <none>
I20260809 22:38:13.533715 7592 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:13.533819 7592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:13.534260 7592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:13.537609 7592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "058679d0b62348fb91beaf86664bf3ed"
format_stamp: "Formatted at 2026-08-09 22:38:13 on dist-test-slave-rgb1"
server_key: "3d3181b8ff2bc3fb5e015ba73f337341"
server_key_iv: "64de3b5041cf5a2188e29b9b81b8423d"
server_key_version: "encryptionkey@0"
I20260809 22:38:13.538529 7592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "058679d0b62348fb91beaf86664bf3ed"
format_stamp: "Formatted at 2026-08-09 22:38:13 on dist-test-slave-rgb1"
server_key: "3d3181b8ff2bc3fb5e015ba73f337341"
server_key_iv: "64de3b5041cf5a2188e29b9b81b8423d"
server_key_version: "encryptionkey@0"
I20260809 22:38:13.544533 7592 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:38:13.548660 7607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:13.550192 7592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 22:38:13.550395 7592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "058679d0b62348fb91beaf86664bf3ed"
format_stamp: "Formatted at 2026-08-09 22:38:13 on dist-test-slave-rgb1"
server_key: "3d3181b8ff2bc3fb5e015ba73f337341"
server_key_iv: "64de3b5041cf5a2188e29b9b81b8423d"
server_key_version: "encryptionkey@0"
I20260809 22:38:13.550643 7592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:13.568365 7592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:13.569381 7592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:13.569720 7592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:13.570771 7592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:13.572337 7592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:13.572466 7592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:13.572639 7592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:13.572741 7592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:13.612540 7451 heartbeater.cc:503] Master 127.28.243.126:36645 was elected leader, sending a full tablet report...
I20260809 22:38:13.612829 7592 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:33061
I20260809 22:38:13.612861 7731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:33061 every 8 connection(s)
I20260809 22:38:13.614454 7592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:38:13.624642 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 7592
I20260809 22:38:13.624835 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:38:13.625300 29645 external_mini_cluster.cc:1428] Setting key 171bab92d501e9d1742b718d1519596b
I20260809 22:38:13.629060 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:13.617374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:50483 (local address 127.28.243.126:36645)
0809 22:38:13.617868 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:13.617875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:13.619160 (+ 1285us) server_negotiation.cc:407] Connection header received
0809 22:38:13.621065 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:13.621070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:13.621139 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:13.621254 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:13.622530 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.623534 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.625556 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.626115 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.626285 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:13.627806 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:13.627822 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:13.627827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:13.627852 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:13.627910 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:13.627935 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:13.628021 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:13.628333 (+ 312us) server_negotiation.cc:299] Negotiation successful
0809 22:38:13.628508 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":250,"threads_started":1}
I20260809 22:38:13.630074 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:13.617939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36645 (local address 127.28.243.67:50483)
0809 22:38:13.618984 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:13.619037 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:13.620789 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:13.621497 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:13.621506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:13.621658 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:13.622314 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:13.622331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.623807 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.623832 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:13.625373 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:13.625407 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.626341 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.626346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:13.626652 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:13.627487 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:38:13.627490 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:13.627532 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:38:13.627614 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:13.628065 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:13.628078 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:13.628090 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 22:38:13.628296 (+ 206us) client_negotiation.cc:240] Negotiation successful
0809 22:38:13.628792 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":345,"threads_started":1}
I20260809 22:38:13.631640 7732 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36645
I20260809 22:38:13.631981 7732 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:13.632956 7732 heartbeater.cc:511] Master 127.28.243.126:36645 requested a full tablet report, sending...
I20260809 22:38:13.634590 7251 ts_manager.cc:194] Registered new tserver with Master: 058679d0b62348fb91beaf86664bf3ed (127.28.243.67:33061)
I20260809 22:38:13.635526 7251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.67:50483
I20260809 22:38:13.641882 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 22:38:13.648707 7734 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:13.646691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.28.243.126:36645)
0809 22:38:13.646925 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:13.646935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:13.647460 (+ 525us) server_negotiation.cc:407] Connection header received
0809 22:38:13.647626 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:13.647630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:13.647697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:13.647787 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:13.648363 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41752: BlockingRecv error: recv got EOF from 127.0.0.1:41752 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:38:13.649426 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 7316
I20260809 22:38:13.659162 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:13.650814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.28.243.65:45271)
0809 22:38:13.651564 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:13.651572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:13.651600 (+ 28us) server_negotiation.cc:407] Connection header received
0809 22:38:13.651671 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:13.651678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:13.651903 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:13.652048 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:13.652474 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.653818 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.655298 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.655978 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.656107 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:13.657522 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:13.657532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:13.657545 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:13.658164 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:13.658212 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:13.658233 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:13.658334 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:13.658587 (+ 253us) server_negotiation.cc:299] Negotiation successful
0809 22:38:13.658714 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":418,"threads_started":1}
I20260809 22:38:13.660655 7445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:13.746667 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 7316
I20260809 22:38:13.765872 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 7455
I20260809 22:38:13.776217 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:13.767134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.28.243.66:43817)
0809 22:38:13.768015 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:13.768023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:13.768054 (+ 31us) server_negotiation.cc:407] Connection header received
0809 22:38:13.768119 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:13.768126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:13.768381 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:13.768545 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:13.769163 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.770241 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.771724 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.772396 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.772512 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:13.774156 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:13.774165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:13.774179 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:13.774863 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:13.774914 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:13.774935 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:13.775054 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:13.775415 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 22:38:13.775606 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":450,"threads_started":1}
I20260809 22:38:13.777872 7581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:13.867291 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 7455
I20260809 22:38:13.889000 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 7592
I20260809 22:38:13.898659 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:13.890299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.28.243.67:33061)
0809 22:38:13.891032 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:13.891040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:13.891074 (+ 34us) server_negotiation.cc:407] Connection header received
0809 22:38:13.891139 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:13.891145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:13.891371 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:13.891590 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:13.892104 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.893322 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.894706 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:13.895327 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:13.895477 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:13.896945 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:13.896957 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:13.896972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:13.897663 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:13.897722 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:13.897744 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:13.897895 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:13.898072 (+ 177us) server_negotiation.cc:299] Negotiation successful
0809 22:38:13.898246 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":402,"threads_started":1}
I20260809 22:38:13.900075 7727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:13.986107 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 7592
I20260809 22:38:14.005230 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 7214
I20260809 22:38:14.006596 7283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:14.115168 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 7214
2026-08-09T22:38:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2819 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T22:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:38:14Z Disabled control of system clock
I20260809 22:38:14.210158 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38137
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:38137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 22:38:14.575062 7754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:14.575398 7754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:14.575493 7754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:14.583631 7754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:38:14.583768 7754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:14.583809 7754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:14.583861 7754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:38:14.583909 7754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:38:14.593561 7754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:38137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:14.594980 7754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:14.596946 7754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:14.611539 7761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:14.611594 7763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:14.611737 7760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:14.612682 7754 server_base.cc:1034] running on GCE node
I20260809 22:38:14.613611 7754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:14.615175 7754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:14.616434 7754 hybrid_clock.cc:648] HybridClock initialized: now 1786315094616392 us; error 44 us; skew 500 ppm
I20260809 22:38:14.616966 7754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:14.619632 7754 webserver.cc:460] Webserver started at http://127.28.243.126:44457/ using document root <none> and password file <none>
I20260809 22:38:14.620398 7754 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:14.620543 7754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:14.621151 7754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:14.627174 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "192eb841658d44dbb14dc477c06b4fec"
format_stamp: "Formatted at 2026-08-09 22:38:14 on dist-test-slave-rgb1"
server_key: "f2e316b49ffaa3ce329212f66bb5b8e9"
server_key_iv: "39fbef282efd624786f91708372627f0"
server_key_version: "encryptionkey@0"
I20260809 22:38:14.628087 7754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "192eb841658d44dbb14dc477c06b4fec"
format_stamp: "Formatted at 2026-08-09 22:38:14 on dist-test-slave-rgb1"
server_key: "f2e316b49ffaa3ce329212f66bb5b8e9"
server_key_iv: "39fbef282efd624786f91708372627f0"
server_key_version: "encryptionkey@0"
I20260809 22:38:14.634220 7754 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 22:38:14.638415 7770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:14.639981 7754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 22:38:14.640216 7754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "192eb841658d44dbb14dc477c06b4fec"
format_stamp: "Formatted at 2026-08-09 22:38:14 on dist-test-slave-rgb1"
server_key: "f2e316b49ffaa3ce329212f66bb5b8e9"
server_key_iv: "39fbef282efd624786f91708372627f0"
server_key_version: "encryptionkey@0"
I20260809 22:38:14.640488 7754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:14.646979 7754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:14.647858 7754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:14.648334 7754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:14.671837 7754 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:38137
I20260809 22:38:14.672101 7828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:38137 every 8 connection(s)
I20260809 22:38:14.673386 7754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:38:14.677838 7829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:38:14.681911 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 7754
I20260809 22:38:14.682178 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:38:14.683063 29645 external_mini_cluster.cc:1428] Setting key d8c93c9eb5d089e418b838dc419f92c3
I20260809 22:38:14.690848 7829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec: Bootstrap starting.
I20260809 22:38:14.693359 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:14.685396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.28.243.126:38137)
0809 22:38:14.686508 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:14.686520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:14.686565 (+ 45us) server_negotiation.cc:407] Connection header received
0809 22:38:14.689310 (+ 2745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:14.689338 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:14.689813 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:14.690191 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:14.691516 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:14.691523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:14.691538 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:14.691556 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:14.691624 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:14.691642 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:14.691833 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:14.691960 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 22:38:14.692276 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":395,"threads_started":1}
I20260809 22:38:14.694473 7829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec: Neither blocks nor log segments found. Creating new log.
I20260809 22:38:14.695873 7829 log.cc:824] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec: Log is configured to *not* fsync() on all Append() calls
I20260809 22:38:14.698839 7829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec: No bootstrap required, opened a new log
I20260809 22:38:14.707290 7829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192eb841658d44dbb14dc477c06b4fec" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38137 } }
I20260809 22:38:14.707685 7829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:38:14.707808 7829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192eb841658d44dbb14dc477c06b4fec, State: Initialized, Role: FOLLOWER
I20260809 22:38:14.708498 7829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192eb841658d44dbb14dc477c06b4fec" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38137 } }
I20260809 22:38:14.708884 7829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:38:14.709069 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:38:14.709285 7829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:38:14.711683 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192eb841658d44dbb14dc477c06b4fec" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38137 } }
I20260809 22:38:14.712195 7829 leader_election.cc:302] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 192eb841658d44dbb14dc477c06b4fec; no voters:
I20260809 22:38:14.712965 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:38:14.713143 7834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:38:14.714422 7834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [term 1 LEADER]: Becoming Leader. State: Replica: 192eb841658d44dbb14dc477c06b4fec, State: Running, Role: LEADER
I20260809 22:38:14.715032 7834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192eb841658d44dbb14dc477c06b4fec" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38137 } }
I20260809 22:38:14.715689 7829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:38:14.718500 7836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 192eb841658d44dbb14dc477c06b4fec. Latest consensus state: current_term: 1 leader_uuid: "192eb841658d44dbb14dc477c06b4fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192eb841658d44dbb14dc477c06b4fec" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38137 } } }
I20260809 22:38:14.718935 7836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [sys.catalog]: This master's current role is: LEADER
I20260809 22:38:14.719373 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "192eb841658d44dbb14dc477c06b4fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "192eb841658d44dbb14dc477c06b4fec" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38137 } } }
I20260809 22:38:14.719805 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec [sys.catalog]: This master's current role is: LEADER
I20260809 22:38:14.721819 7842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:38:14.728485 7842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:38:14.739300 7842 catalog_manager.cc:1347] Generated new cluster ID: 911acd8f0cf1429a8d7b6c04d331aa8d
I20260809 22:38:14.739454 7842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:38:14.781904 7842 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:38:14.782481 7842 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:38:14.792085 7842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 192eb841658d44dbb14dc477c06b4fec: Generated new TSK 0
I20260809 22:38:14.793241 7842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:38:14.812674 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38137
--builtin_ntp_servers=127.28.243.84:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 22:38:15.234454 7853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:15.234822 7853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:15.234987 7853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:15.242913 7853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:15.243126 7853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:15.243343 7853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:38:15.256291 7853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:15.258011 7853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:15.260018 7853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:15.275156 7861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:15.275293 7859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:15.275135 7858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:15.276830 7853 server_base.cc:1034] running on GCE node
I20260809 22:38:15.277706 7853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:15.279801 7853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:15.281059 7853 hybrid_clock.cc:648] HybridClock initialized: now 1786315095280992 us; error 69 us; skew 500 ppm
I20260809 22:38:15.281416 7853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:15.283978 7853 webserver.cc:460] Webserver started at http://127.28.243.65:39809/ using document root <none> and password file <none>
I20260809 22:38:15.284797 7853 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:15.284900 7853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:15.285249 7853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:15.288491 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "d81ccfc94ee4470b97000245aae8e2d8"
format_stamp: "Formatted at 2026-08-09 22:38:15 on dist-test-slave-rgb1"
server_key: "61dae410504cc107f3b78130eb1c1af8"
server_key_iv: "2d4f72b533d32ff3ffb13555efac117f"
server_key_version: "encryptionkey@0"
I20260809 22:38:15.289623 7853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "d81ccfc94ee4470b97000245aae8e2d8"
format_stamp: "Formatted at 2026-08-09 22:38:15 on dist-test-slave-rgb1"
server_key: "61dae410504cc107f3b78130eb1c1af8"
server_key_iv: "2d4f72b533d32ff3ffb13555efac117f"
server_key_version: "encryptionkey@0"
I20260809 22:38:15.295779 7853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 22:38:15.299988 7867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:15.301910 7853 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:38:15.302162 7853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "d81ccfc94ee4470b97000245aae8e2d8"
format_stamp: "Formatted at 2026-08-09 22:38:15 on dist-test-slave-rgb1"
server_key: "61dae410504cc107f3b78130eb1c1af8"
server_key_iv: "2d4f72b533d32ff3ffb13555efac117f"
server_key_version: "encryptionkey@0"
I20260809 22:38:15.302436 7853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:15.309314 7853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:15.310230 7853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:15.310527 7853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:15.311564 7853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:15.313233 7853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:15.313369 7853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:15.313516 7853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:15.313614 7853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:15.354573 7853 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:37241
I20260809 22:38:15.354669 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:37241 every 8 connection(s)
I20260809 22:38:15.355950 7853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:38:15.359817 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 7853
I20260809 22:38:15.360046 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:38:15.360807 29645 external_mini_cluster.cc:1428] Setting key 4bf0ce3a7a66eb2dd99dab1ac13630d2
I20260809 22:38:15.364159 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38137
--builtin_ntp_servers=127.28.243.84:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 22:38:15.366843 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:15.359091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:59513 (local address 127.28.243.126:38137)
0809 22:38:15.359691 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:15.359700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:15.361399 (+ 1699us) server_negotiation.cc:407] Connection header received
0809 22:38:15.364038 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:15.364045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:15.364122 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:15.364282 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:15.365928 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:15.365935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:15.365941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:15.365963 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:15.366021 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:15.366034 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:15.366194 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:15.366427 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 22:38:15.366508 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
I20260809 22:38:15.367919 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:15.359789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38137 (local address 127.28.243.65:59513)
0809 22:38:15.361163 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:15.361218 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:15.363661 (+ 2443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:15.364631 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:15.364650 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:15.364763 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:15.365577 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:38:15.365580 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:15.365631 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:38:15.365727 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:15.366153 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:15.366168 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:15.366178 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 22:38:15.366531 (+ 353us) client_negotiation.cc:240] Negotiation successful
0809 22:38:15.366840 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":455,"threads_started":1}
I20260809 22:38:15.369851 7988 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38137
I20260809 22:38:15.370263 7988 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:15.370927 7988 heartbeater.cc:511] Master 127.28.243.126:38137 requested a full tablet report, sending...
I20260809 22:38:15.373980 7787 ts_manager.cc:194] Registered new tserver with Master: d81ccfc94ee4470b97000245aae8e2d8 (127.28.243.65:37241)
W20260809 22:38:15.729169 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:15.729593 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:15.729791 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:15.738752 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:15.738973 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:15.739189 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:38:15.750260 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:15.751968 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:15.754415 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:15.771301 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:15.771359 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:15.771622 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:15.775076 7992 server_base.cc:1034] running on GCE node
I20260809 22:38:15.775880 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:15.777276 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:15.778569 7992 hybrid_clock.cc:648] HybridClock initialized: now 1786315095778510 us; error 54 us; skew 500 ppm
I20260809 22:38:15.778975 7992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:15.781693 7992 webserver.cc:460] Webserver started at http://127.28.243.66:39175/ using document root <none> and password file <none>
I20260809 22:38:15.782450 7992 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:15.782598 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:15.782943 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:15.786130 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "337b86ae8224493f8e6d08acdf4fa9b0"
format_stamp: "Formatted at 2026-08-09 22:38:15 on dist-test-slave-rgb1"
server_key: "51423de09b1ad6f56e2fa361c6bf4905"
server_key_iv: "ad0d2ca733721c8f90762c239793e877"
server_key_version: "encryptionkey@0"
I20260809 22:38:15.786967 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "337b86ae8224493f8e6d08acdf4fa9b0"
format_stamp: "Formatted at 2026-08-09 22:38:15 on dist-test-slave-rgb1"
server_key: "51423de09b1ad6f56e2fa361c6bf4905"
server_key_iv: "ad0d2ca733721c8f90762c239793e877"
server_key_version: "encryptionkey@0"
I20260809 22:38:15.792644 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260809 22:38:15.796221 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:15.797994 7992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 22:38:15.798199 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "337b86ae8224493f8e6d08acdf4fa9b0"
format_stamp: "Formatted at 2026-08-09 22:38:15 on dist-test-slave-rgb1"
server_key: "51423de09b1ad6f56e2fa361c6bf4905"
server_key_iv: "ad0d2ca733721c8f90762c239793e877"
server_key_version: "encryptionkey@0"
I20260809 22:38:15.798457 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:15.805366 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:15.806273 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:15.806602 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:15.807783 7992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:15.809590 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:15.809692 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:15.809875 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:15.809991 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:15.850041 7992 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:34837
I20260809 22:38:15.850107 8128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:34837 every 8 connection(s)
I20260809 22:38:15.851611 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:38:15.858440 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 7992
I20260809 22:38:15.858682 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:38:15.859179 29645 external_mini_cluster.cc:1428] Setting key 7b6817cab130fcdf4405894bec95632f
I20260809 22:38:15.861341 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:15.854625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:44725 (local address 127.28.243.126:38137)
0809 22:38:15.854968 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:15.854975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:15.856479 (+ 1504us) server_negotiation.cc:407] Connection header received
0809 22:38:15.858634 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:15.858643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:15.858726 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:15.858977 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:15.860484 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:15.860491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:15.860496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:15.860516 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:15.860605 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:15.860621 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:15.860806 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:15.861008 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 22:38:15.861080 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260809 22:38:15.862274 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38137
--builtin_ntp_servers=127.28.243.84:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 22:38:15.862603 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:15.855241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38137 (local address 127.28.243.66:44725)
0809 22:38:15.856298 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:15.856339 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:15.858249 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:15.859209 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:15.859218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:15.859384 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:15.860167 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:38:15.860169 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:15.860208 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:38:15.860296 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:15.860751 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:15.860764 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:15.860775 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 22:38:15.861098 (+ 323us) client_negotiation.cc:240] Negotiation successful
0809 22:38:15.861382 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":325,"threads_started":1}
I20260809 22:38:15.864274 8129 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38137
I20260809 22:38:15.864717 8129 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:15.865363 8129 heartbeater.cc:511] Master 127.28.243.126:38137 requested a full tablet report, sending...
I20260809 22:38:15.867048 7787 ts_manager.cc:194] Registered new tserver with Master: 337b86ae8224493f8e6d08acdf4fa9b0 (127.28.243.66:34837)
W20260809 22:38:16.218034 8132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:16.218408 8132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:16.218580 8132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:16.226275 8132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:16.226441 8132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:16.226640 8132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:38:16.236196 8132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:16.237618 8132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:16.239461 8132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:16.253810 8138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:16.253755 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:16.255259 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:16.255589 8132 server_base.cc:1034] running on GCE node
I20260809 22:38:16.256285 8132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:16.257421 8132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:16.258656 8132 hybrid_clock.cc:648] HybridClock initialized: now 1786315096258602 us; error 58 us; skew 500 ppm
I20260809 22:38:16.259025 8132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:38:16.261643 8132 webserver.cc:460] Webserver started at http://127.28.243.67:35025/ using document root <none> and password file <none>
I20260809 22:38:16.262342 8132 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:16.262480 8132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:16.262852 8132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:16.266120 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "c64b22ec4f214ee4859268f8df538e60"
format_stamp: "Formatted at 2026-08-09 22:38:16 on dist-test-slave-rgb1"
server_key: "558efb647b96f5c9babb8620cf79fba9"
server_key_iv: "c4ac7dfbd7e4c295e83394963b1f0d71"
server_key_version: "encryptionkey@0"
I20260809 22:38:16.266923 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "c64b22ec4f214ee4859268f8df538e60"
format_stamp: "Formatted at 2026-08-09 22:38:16 on dist-test-slave-rgb1"
server_key: "558efb647b96f5c9babb8620cf79fba9"
server_key_iv: "c4ac7dfbd7e4c295e83394963b1f0d71"
server_key_version: "encryptionkey@0"
I20260809 22:38:16.273154 8132 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:38:16.277899 8146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:16.279654 8132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260809 22:38:16.279815 8132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "c64b22ec4f214ee4859268f8df538e60"
format_stamp: "Formatted at 2026-08-09 22:38:16 on dist-test-slave-rgb1"
server_key: "558efb647b96f5c9babb8620cf79fba9"
server_key_iv: "c4ac7dfbd7e4c295e83394963b1f0d71"
server_key_version: "encryptionkey@0"
I20260809 22:38:16.280030 8132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:16.286347 8132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:16.287189 8132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:16.287443 8132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:16.288488 8132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:16.290632 8132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:16.290740 8132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:16.290904 8132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:16.291021 8132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:16.329211 8132 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:35945
I20260809 22:38:16.329310 8266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:35945 every 8 connection(s)
I20260809 22:38:16.330642 8132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:38:16.334319 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 8132
I20260809 22:38:16.334545 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:38:16.334995 29645 external_mini_cluster.cc:1428] Setting key 7fa4d14e51bcdfe39091ac0ae553d183
I20260809 22:38:16.341224 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:16.333672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:43879 (local address 127.28.243.126:38137)
0809 22:38:16.333862 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:16.333867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:16.336452 (+ 2585us) server_negotiation.cc:407] Connection header received
0809 22:38:16.338769 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:16.338773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:16.338841 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:16.338964 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:16.340398 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:16.340403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:16.340407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:16.340425 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:16.340462 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:16.340475 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:16.340682 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:16.340891 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 22:38:16.340967 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:38:16.342121 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:16.334181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38137 (local address 127.28.243.67:43879)
0809 22:38:16.336272 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:16.336353 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:16.338485 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:16.339173 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:16.339182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:16.339275 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:16.340050 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:38:16.340052 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:16.340092 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:38:16.340182 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:16.340657 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:16.340665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:16.340671 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 22:38:16.340907 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 22:38:16.341114 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"thread_start_us":1330,"threads_started":1}
I20260809 22:38:16.343729 8267 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38137
I20260809 22:38:16.344154 8267 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:16.345009 8267 heartbeater.cc:511] Master 127.28.243.126:38137 requested a full tablet report, sending...
I20260809 22:38:16.346660 7787 ts_manager.cc:194] Registered new tserver with Master: c64b22ec4f214ee4859268f8df538e60 (127.28.243.67:35945)
I20260809 22:38:16.351030 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 22:38:16.358173 7990 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:38:16.356443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.28.243.126:38137)
0809 22:38:16.356687 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:16.356695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:16.357247 (+ 552us) server_negotiation.cc:407] Connection header received
0809 22:38:16.357382 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:16.357388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:16.357456 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:16.357585 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:16.357910 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43340: BlockingRecv error: recv got EOF from 127.0.0.1:43340 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 22:38:16.358593 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 7853
I20260809 22:38:16.362767 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:16.359835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.28.243.65:37241)
0809 22:38:16.360450 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:16.360459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:16.360492 (+ 33us) server_negotiation.cc:407] Connection header received
0809 22:38:16.360589 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:16.360597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:16.360827 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:16.360952 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:16.361875 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:16.361881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:16.361891 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:16.361904 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:16.362045 (+ 141us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:16.362066 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:16.362200 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:16.362300 (+ 100us) server_negotiation.cc:299] Negotiation successful
0809 22:38:16.362385 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":303,"threads_started":1}
I20260809 22:38:16.364277 7983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:16.451102 7988 heartbeater.cc:503] Master 127.28.243.126:38137 was elected leader, sending a full tablet report...
I20260809 22:38:16.452618 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 7853
W20260809 22:38:16.474145 7772 connection.cc:535] server connection from 127.28.243.65:59513 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 22:38:16.474555 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 7992
I20260809 22:38:16.478718 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:16.475532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.28.243.66:34837)
0809 22:38:16.476122 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:16.476129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:16.476159 (+ 30us) server_negotiation.cc:407] Connection header received
0809 22:38:16.476224 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:16.476231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:16.476460 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:16.476684 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:16.477777 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:16.477785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:16.477797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:16.477814 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:16.477879 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:16.477904 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:16.478028 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:16.478289 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 22:38:16.478430 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":249,"threads_started":1}
I20260809 22:38:16.480157 8124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:16.564769 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 7992
I20260809 22:38:16.584746 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 8132
I20260809 22:38:16.589840 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:16.585887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.28.243.67:35945)
0809 22:38:16.586751 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:16.586758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:16.586794 (+ 36us) server_negotiation.cc:407] Connection header received
0809 22:38:16.586868 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:16.586875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:16.587102 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:16.587246 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:16.588453 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:16.588460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:16.588472 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:38:16.588492 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:16.588535 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:38:16.588550 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:16.589354 (+ 804us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:16.589440 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 22:38:16.589535 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":515,"threads_started":1}
I20260809 22:38:16.591298 8261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:16.678805 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 8132
I20260809 22:38:16.697554 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 7754
I20260809 22:38:16.698982 7822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:16.808748 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 7754
2026-08-09T22:38:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2692 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:38:16 dist-test-slave-rgb1 krb5kdc[8287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:38:16 dist-test-slave-rgb1 krb5kdc[8287](info): set up 2 sockets
Aug 09 22:38:16 dist-test-slave-rgb1 krb5kdc[8287](info): commencing operation
krb5kdc: starting...
W20260809 22:38:18.088599 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:38:18 dist-test-slave-rgb1 krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:38:18.419176 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33157
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:38:18.792173 8300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:18.792615 8300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:18.792800 8300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:18.802525 8300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:38:18.802742 8300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:18.802856 8300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:18.802969 8300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:38:18.803052 8300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:38:18.815555 8300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:18.817404 8300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:18.818899 8300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:18.833317 8305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:18.833763 8308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:18.833796 8306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:18.835031 8300 server_base.cc:1034] running on GCE node
I20260809 22:38:18.836048 8300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:18.837739 8300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:18.839191 8300 hybrid_clock.cc:648] HybridClock initialized: now 1786315098839074 us; error 99 us; skew 500 ppm
I20260809 22:38:18.839663 8300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:38:18 dist-test-slave-rgb1 krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315098, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:38:18.843386 8300 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:38:18.845208 8300 webserver.cc:460] Webserver started at http://127.28.243.126:42977/ using document root <none> and password file <none>
I20260809 22:38:18.845920 8300 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:18.846035 8300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:18.846319 8300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:18.849618 8300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "f0ed3385ba3744e491199902bef8d15d"
format_stamp: "Formatted at 2026-08-09 22:38:18 on dist-test-slave-rgb1"
server_key: "5b8277cc05d63907283d9da782a1bd5f"
server_key_iv: "4f3995f5910727754381cd535961ac53"
server_key_version: "encryptionkey@0"
I20260809 22:38:18.850389 8300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "f0ed3385ba3744e491199902bef8d15d"
format_stamp: "Formatted at 2026-08-09 22:38:18 on dist-test-slave-rgb1"
server_key: "5b8277cc05d63907283d9da782a1bd5f"
server_key_iv: "4f3995f5910727754381cd535961ac53"
server_key_version: "encryptionkey@0"
I20260809 22:38:18.856194 8300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 22:38:18.860425 8315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:18.862402 8300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:38:18.862643 8300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "f0ed3385ba3744e491199902bef8d15d"
format_stamp: "Formatted at 2026-08-09 22:38:18 on dist-test-slave-rgb1"
server_key: "5b8277cc05d63907283d9da782a1bd5f"
server_key_iv: "4f3995f5910727754381cd535961ac53"
server_key_version: "encryptionkey@0"
I20260809 22:38:18.862895 8300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:18.883534 8300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:18.888900 8300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:18.889261 8300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:18.911309 8300 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33157
I20260809 22:38:18.911355 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33157 every 8 connection(s)
I20260809 22:38:18.912927 8300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:38:18.917009 8369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:38:18.922461 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 8300
I20260809 22:38:18.922731 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:38:18.923552 29645 external_mini_cluster.cc:1428] Setting key 71a85de62ffc132d0217b78da88b9775
I20260809 22:38:18.929754 8369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d: Bootstrap starting.
I20260809 22:38:18.934057 8369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d: Neither blocks nor log segments found. Creating new log.
I20260809 22:38:18.935513 8369 log.cc:824] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:38:18.939704 8369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d: No bootstrap required, opened a new log
Aug 09 22:38:18 dist-test-slave-rgb1 krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:38:18.951298 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:18.925921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.28.243.126:33157)
0809 22:38:18.926913 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:18.926925 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:18.926977 (+ 52us) server_negotiation.cc:407] Connection header received
0809 22:38:18.928899 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:18.928924 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:18.929349 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:18.929717 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:18.931950 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:18.933410 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:18.935400 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:18.936435 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:18.936707 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:18.941653 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:18.941667 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:18.941698 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:18.941752 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:18.946200 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:18.946955 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:18.946964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:18.946978 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:18.947206 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:18.947591 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:18.947600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:18.947605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:18.948452 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:18.948844 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:18.949298 (+ 454us) server_negotiation.cc:299] Negotiation successful
0809 22:38:18.949817 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":458,"threads_started":1}
I20260809 22:38:18.951123 8369 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ed3385ba3744e491199902bef8d15d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33157 } }
I20260809 22:38:18.951618 8369 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:38:18.951773 8369 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0ed3385ba3744e491199902bef8d15d, State: Initialized, Role: FOLLOWER
I20260809 22:38:18.952389 8369 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ed3385ba3744e491199902bef8d15d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33157 } }
I20260809 22:38:18.952693 8369 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:38:18.952849 8369 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:38:18.953033 8369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:38:18.956128 8369 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ed3385ba3744e491199902bef8d15d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33157 } }
I20260809 22:38:18.956719 8369 leader_election.cc:302] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0ed3385ba3744e491199902bef8d15d; no voters:
I20260809 22:38:18.957345 8369 leader_election.cc:288] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:38:18.957607 8374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:38:18.958528 8374 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [term 1 LEADER]: Becoming Leader. State: Replica: f0ed3385ba3744e491199902bef8d15d, State: Running, Role: LEADER
I20260809 22:38:18.959390 8374 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ed3385ba3744e491199902bef8d15d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33157 } }
I20260809 22:38:18.959728 8369 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:38:18.962780 8375 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0ed3385ba3744e491199902bef8d15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ed3385ba3744e491199902bef8d15d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33157 } } }
I20260809 22:38:18.963215 8375 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [sys.catalog]: This master's current role is: LEADER
I20260809 22:38:18.963806 8376 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0ed3385ba3744e491199902bef8d15d. Latest consensus state: current_term: 1 leader_uuid: "f0ed3385ba3744e491199902bef8d15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0ed3385ba3744e491199902bef8d15d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33157 } } }
I20260809 22:38:18.964291 8376 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d [sys.catalog]: This master's current role is: LEADER
I20260809 22:38:18.964491 8381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:38:18.982547 8381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:38:18.992875 8381 catalog_manager.cc:1347] Generated new cluster ID: 2b18b34591524f7bb7fabdb30722d8a3
I20260809 22:38:18.993126 8381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:38:19.027820 8381 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:38:19.029894 8381 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:38:19.043108 8381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d: Generated new TSK 0
I20260809 22:38:19.044284 8381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:38:19.129818 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33157
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:38:19.507774 8396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:19.508193 8396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:19.508347 8396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:19.515990 8396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:19.516111 8396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:19.516228 8396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:38:19.526304 8396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:19.527681 8396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:19.529052 8396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:19.543562 8401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:19.543603 8404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:19.543641 8402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:19.545058 8396 server_base.cc:1034] running on GCE node
I20260809 22:38:19.545782 8396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:19.546811 8396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:19.548070 8396 hybrid_clock.cc:648] HybridClock initialized: now 1786315099548010 us; error 43 us; skew 500 ppm
I20260809 22:38:19.548440 8396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:38:19 dist-test-slave-rgb1 krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:38:19.552129 8396 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:38:19.553763 8396 webserver.cc:460] Webserver started at http://127.28.243.65:35257/ using document root <none> and password file <none>
I20260809 22:38:19.554566 8396 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:19.554713 8396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:19.555028 8396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:19.558282 8396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "b8e48f8c5e4b433da076490151da0136"
format_stamp: "Formatted at 2026-08-09 22:38:19 on dist-test-slave-rgb1"
server_key: "4ae76a302d2aff5e61a5772534138a49"
server_key_iv: "eb3fb63341b195ab67d25fcb3007dda1"
server_key_version: "encryptionkey@0"
I20260809 22:38:19.559159 8396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "b8e48f8c5e4b433da076490151da0136"
format_stamp: "Formatted at 2026-08-09 22:38:19 on dist-test-slave-rgb1"
server_key: "4ae76a302d2aff5e61a5772534138a49"
server_key_iv: "eb3fb63341b195ab67d25fcb3007dda1"
server_key_version: "encryptionkey@0"
I20260809 22:38:19.565034 8396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 22:38:19.569270 8412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:19.571075 8396 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:38:19.571305 8396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "b8e48f8c5e4b433da076490151da0136"
format_stamp: "Formatted at 2026-08-09 22:38:19 on dist-test-slave-rgb1"
server_key: "4ae76a302d2aff5e61a5772534138a49"
server_key_iv: "eb3fb63341b195ab67d25fcb3007dda1"
server_key_version: "encryptionkey@0"
I20260809 22:38:19.571549 8396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:19.592054 8396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:19.597101 8396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:19.597484 8396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:19.598690 8396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:19.600430 8396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:19.600548 8396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:19.600754 8396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:19.600843 8396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:19.640875 8396 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:43251
I20260809 22:38:19.640937 8527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:43251 every 8 connection(s)
I20260809 22:38:19.642392 8396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:38:19.649677 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 8396
I20260809 22:38:19.649976 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:38:19.650669 29645 external_mini_cluster.cc:1428] Setting key 60cd401a0700d5744b8f5d0f1e39a063
Aug 09 22:38:19 dist-test-slave-rgb1 krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315099, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:38:19.665534 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:19.645573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:42503 (local address 127.28.243.126:33157)
0809 22:38:19.646039 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:19.646046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:19.647426 (+ 1380us) server_negotiation.cc:407] Connection header received
0809 22:38:19.649622 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:19.649628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:19.649733 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:19.649866 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:19.651697 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:19.652531 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:19.654343 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:19.654794 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:19.654937 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:19.659102 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:19.659107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:19.659111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:19.659147 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:19.662095 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:19.662920 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:19.662929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:19.662934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:19.663076 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:19.663551 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:19.663556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:19.663559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:19.663856 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:19.664029 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:19.665040 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0809 22:38:19.665261 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260809 22:38:19.667069 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:19.646308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33157 (local address 127.28.243.65:42503)
0809 22:38:19.647238 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:19.647286 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:19.649277 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:19.650178 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:19.650188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:19.650840 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:19.651507 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:19.651526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:19.652770 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:19.652774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:19.654165 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:19.654179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:19.655065 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:19.655073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:19.655326 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:19.656142 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:19.656173 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:19.658827 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:19.662265 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:38:19.662276 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:38:19.662298 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:38:19.662783 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:38:19.663196 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:38:19.663201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:38:19.663205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:38:19.663424 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:38:19.663999 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:19.664008 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:19.664460 (+ 452us) client_negotiation.cc:769] Sending connection context
0809 22:38:19.664868 (+ 408us) client_negotiation.cc:240] Negotiation successful
0809 22:38:19.665542 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":392,"threads_started":1}
I20260809 22:38:19.669106 8528 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33157
I20260809 22:38:19.669461 8528 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:19.670490 8528 heartbeater.cc:511] Master 127.28.243.126:33157 requested a full tablet report, sending...
I20260809 22:38:19.672969 8333 ts_manager.cc:194] Registered new tserver with Master: b8e48f8c5e4b433da076490151da0136 (127.28.243.65:43251)
I20260809 22:38:19.674440 8333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:42503
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:38:19.726186 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33157
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:38:20.105507 8534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:20.105835 8534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:20.106009 8534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:20.113817 8534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:20.114010 8534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:20.114190 8534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:38:20.127444 8534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:20.129101 8534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:20.130609 8534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:20.147598 8539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:20.147568 8540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:20.147625 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:20.148469 8534 server_base.cc:1034] running on GCE node
I20260809 22:38:20.149252 8534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:20.150355 8534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:20.151602 8534 hybrid_clock.cc:648] HybridClock initialized: now 1786315100151525 us; error 59 us; skew 500 ppm
I20260809 22:38:20.152015 8534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:38:20 dist-test-slave-rgb1 krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:38:20.160076 8534 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:38:20.162009 8534 webserver.cc:460] Webserver started at http://127.28.243.66:38363/ using document root <none> and password file <none>
I20260809 22:38:20.163074 8534 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:20.163277 8534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:20.163643 8534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:20.167081 8534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "cb942da9e1cc448f855cf709efd10406"
format_stamp: "Formatted at 2026-08-09 22:38:20 on dist-test-slave-rgb1"
server_key: "cfbc7fd2bf997629e4904a786cfeea77"
server_key_iv: "c0332237aeecb7a43c9d237b7f47a943"
server_key_version: "encryptionkey@0"
I20260809 22:38:20.167943 8534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "cb942da9e1cc448f855cf709efd10406"
format_stamp: "Formatted at 2026-08-09 22:38:20 on dist-test-slave-rgb1"
server_key: "cfbc7fd2bf997629e4904a786cfeea77"
server_key_iv: "c0332237aeecb7a43c9d237b7f47a943"
server_key_version: "encryptionkey@0"
I20260809 22:38:20.174377 8534 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20260809 22:38:20.179070 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:20.180907 8534 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 22:38:20.181123 8534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "cb942da9e1cc448f855cf709efd10406"
format_stamp: "Formatted at 2026-08-09 22:38:20 on dist-test-slave-rgb1"
server_key: "cfbc7fd2bf997629e4904a786cfeea77"
server_key_iv: "c0332237aeecb7a43c9d237b7f47a943"
server_key_version: "encryptionkey@0"
I20260809 22:38:20.181363 8534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:20.208158 8534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:20.213939 8534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:20.214296 8534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:20.215392 8534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:20.216974 8534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:20.217062 8534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:20.217186 8534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:20.217247 8534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:20.252750 8534 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:46559
I20260809 22:38:20.252802 8670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:46559 every 8 connection(s)
I20260809 22:38:20.254070 8534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:38:20.261332 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 8534
I20260809 22:38:20.261570 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:38:20.262009 29645 external_mini_cluster.cc:1428] Setting key e59655f895b35c03ceba605246d4c05d
Aug 09 22:38:20 dist-test-slave-rgb1 krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:38:20.279103 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.256930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:44655 (local address 127.28.243.126:33157)
0809 22:38:20.257413 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:20.257418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:20.258489 (+ 1071us) server_negotiation.cc:407] Connection header received
0809 22:38:20.260647 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:20.260656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:20.260746 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:20.260885 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:20.262743 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.263556 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.265260 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.266011 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.266147 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.270758 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:20.270767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:20.270775 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:20.270834 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:20.274096 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:20.275277 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:20.275292 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:20.275294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:20.275542 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:20.276217 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:20.276222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:20.276224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:20.276766 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:20.277058 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:20.278239 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0809 22:38:20.278558 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
I20260809 22:38:20.279945 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.257494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33157 (local address 127.28.243.66:44655)
0809 22:38:20.258294 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:20.258336 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:20.260242 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:20.261145 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:20.261155 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:20.261840 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:20.262542 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.262562 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.263734 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.263739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.265100 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.265113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.266330 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.266340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.266565 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.267345 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:20.267371 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:20.270424 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:20.274283 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:38:20.274297 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:38:20.274321 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:38:20.275145 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:38:20.275685 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:38:20.275693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:38:20.275700 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:38:20.275991 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:38:20.277015 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:20.277027 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:20.277598 (+ 571us) client_negotiation.cc:769] Sending connection context
0809 22:38:20.278024 (+ 426us) client_negotiation.cc:240] Negotiation successful
0809 22:38:20.278596 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":327,"threads_started":1}
I20260809 22:38:20.281673 8671 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33157
I20260809 22:38:20.282032 8671 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:20.282778 8671 heartbeater.cc:511] Master 127.28.243.126:33157 requested a full tablet report, sending...
I20260809 22:38:20.284497 8333 ts_manager.cc:194] Registered new tserver with Master: cb942da9e1cc448f855cf709efd10406 (127.28.243.66:46559)
I20260809 22:38:20.285900 8333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:44655
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:38:20.333343 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33157
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:38:20.678987 8528 heartbeater.cc:503] Master 127.28.243.126:33157 was elected leader, sending a full tablet report...
W20260809 22:38:20.723726 8677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:38:20.724123 8677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:38:20.724502 8677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:38:20.733799 8677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:38:20.733980 8677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:38:20.734189 8677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:38:20.746449 8677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:38:20.747948 8677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:38:20.749436 8677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:38:20.766192 8682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:20.766391 8683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:38:20.766707 8685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:38:20.768122 8677 server_base.cc:1034] running on GCE node
I20260809 22:38:20.768913 8677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:38:20.770074 8677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:38:20.771546 8677 hybrid_clock.cc:648] HybridClock initialized: now 1786315100771484 us; error 50 us; skew 500 ppm
I20260809 22:38:20.771972 8677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:38:20 dist-test-slave-rgb1 krb5kdc[8287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:38:20.775583 8677 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:38:20.777499 8677 webserver.cc:460] Webserver started at http://127.28.243.67:43055/ using document root <none> and password file <none>
I20260809 22:38:20.778260 8677 fs_manager.cc:359] Metadata directory not provided
I20260809 22:38:20.778424 8677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:38:20.778756 8677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:38:20.782064 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "a1d0cc902a79475b874d9f5b9a35a56d"
format_stamp: "Formatted at 2026-08-09 22:38:20 on dist-test-slave-rgb1"
server_key: "01bff03ed5aa18e93ccb6ff991fc0a3b"
server_key_iv: "9ce40743aa9860a5afece4e52dfcdc4a"
server_key_version: "encryptionkey@0"
I20260809 22:38:20.782882 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "a1d0cc902a79475b874d9f5b9a35a56d"
format_stamp: "Formatted at 2026-08-09 22:38:20 on dist-test-slave-rgb1"
server_key: "01bff03ed5aa18e93ccb6ff991fc0a3b"
server_key_iv: "9ce40743aa9860a5afece4e52dfcdc4a"
server_key_version: "encryptionkey@0"
I20260809 22:38:20.788827 8677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:38:20.792811 8692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:20.794385 8677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:38:20.794610 8677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "a1d0cc902a79475b874d9f5b9a35a56d"
format_stamp: "Formatted at 2026-08-09 22:38:20 on dist-test-slave-rgb1"
server_key: "01bff03ed5aa18e93ccb6ff991fc0a3b"
server_key_iv: "9ce40743aa9860a5afece4e52dfcdc4a"
server_key_version: "encryptionkey@0"
I20260809 22:38:20.794816 8677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:38:20.815987 8677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:38:20.820576 8677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:38:20.820842 8677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:38:20.821882 8677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:38:20.823328 8677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:38:20.823407 8677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:20.823499 8677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:38:20.823545 8677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:38:20.863488 8677 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:40279
I20260809 22:38:20.863505 8810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:40279 every 8 connection(s)
I20260809 22:38:20.864920 8677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:38:20.866403 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 8677
I20260809 22:38:20.866641 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:38:20.867219 29645 external_mini_cluster.cc:1428] Setting key 2b95da14ff8032c316e145d3bbd62011
Aug 09 22:38:20 dist-test-slave-rgb1 krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315100, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:38:20.891448 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.868477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:42523 (local address 127.28.243.126:33157)
0809 22:38:20.869478 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:20.869484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:20.870844 (+ 1360us) server_negotiation.cc:407] Connection header received
0809 22:38:20.872916 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:20.872920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:20.873024 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:20.873146 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:20.875592 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.876957 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.878730 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.879230 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.879336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.883701 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:20.883706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:20.883709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:20.883752 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:20.887585 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:20.888425 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:20.888429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:20.888432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:20.888649 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:20.889202 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:20.889208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:20.889211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:20.889730 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:20.889941 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:20.890930 (+ 989us) server_negotiation.cc:299] Negotiation successful
0809 22:38:20.891148 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":714,"threads_started":1}
I20260809 22:38:20.892642 8815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.869521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33157 (local address 127.28.243.67:42523)
0809 22:38:20.870658 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:20.870710 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:20.872635 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:20.873627 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:20.873641 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:20.874474 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:38:20.875265 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.875302 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.877149 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.877154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.878536 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.878550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.879469 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.879476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.879687 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.880478 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:38:20.880513 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:38:20.883427 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:38:20.887747 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:38:20.887755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:38:20.887769 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:38:20.888287 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:38:20.888755 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:38:20.888758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:38:20.888761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:38:20.888986 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:38:20.889974 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:38:20.889981 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:38:20.890454 (+ 473us) client_negotiation.cc:769] Sending connection context
0809 22:38:20.890796 (+ 342us) client_negotiation.cc:240] Negotiation successful
0809 22:38:20.891269 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":425,"threads_started":1}
I20260809 22:38:20.894299 8812 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33157
I20260809 22:38:20.894642 8812 heartbeater.cc:464] Registering TS with master...
I20260809 22:38:20.895623 8812 heartbeater.cc:511] Master 127.28.243.126:33157 requested a full tablet report, sending...
I20260809 22:38:20.897531 8333 ts_manager.cc:194] Registered new tserver with Master: a1d0cc902a79475b874d9f5b9a35a56d (127.28.243.67:40279)
I20260809 22:38:20.898595 8333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:42523
I20260809 22:38:20.910230 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:38:20.930294 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.915618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.28.243.126:33157)
0809 22:38:20.915820 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:20.915829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:20.916317 (+ 488us) server_negotiation.cc:407] Connection header received
0809 22:38:20.916540 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:20.916551 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:20.916725 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:20.916843 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:20.918447 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.919272 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.920703 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.921183 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.921340 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.923853 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:20.923858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:20.923861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:20.923894 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:20.927314 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:20.928064 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:20.928070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:20.928072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:20.928209 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:20.928659 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:20.928665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:20.928668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:20.929072 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:20.929351 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:20.929709 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 22:38:20.929978 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 22:38:20.935582 8333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:38:20.943683 8333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:38:20.970817 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.958966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.28.243.66:46559)
0809 22:38:20.960452 (+ 1486us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:20.960466 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:20.960503 (+ 37us) server_negotiation.cc:407] Connection header received
0809 22:38:20.960642 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:20.960653 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:20.961105 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:20.961483 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:38:20.963395 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.965403 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.967982 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.969339 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.969712 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.969937 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:20.970288 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 22:38:20.970469 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":1159,"threads_started":1}
I20260809 22:38:20.970948 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.958854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:46559 (local address 127.0.0.1:56188)
0809 22:38:20.959942 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:20.959989 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:20.960214 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:20.961925 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:20.961934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:20.961991 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:38:20.962922 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.962944 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.965553 (+ 2609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.965561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.967723 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.967737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.969740 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.969747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.970033 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.970077 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 22:38:20.970225 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 22:38:20.970440 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":26,"thread_start_us":463,"threads_started":1}
I20260809 22:38:20.975680 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.959002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:43251 (local address 127.0.0.1:50612)
0809 22:38:20.963176 (+ 4174us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:20.963197 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:20.963339 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:20.964639 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:20.964645 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:20.964676 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:38:20.965169 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.965189 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.967573 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.967582 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.970903 (+ 3321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.970918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.974706 (+ 3788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.974714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.974911 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.974952 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 22:38:20.975102 (+ 150us) client_negotiation.cc:240] Negotiation successful
0809 22:38:20.975263 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3811,"mutex_wait_us":189,"thread_start_us":3571,"threads_started":1}
I20260809 22:38:20.975728 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.959699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.28.243.65:43251)
0809 22:38:20.962367 (+ 2668us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:20.962380 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:20.963484 (+ 1104us) server_negotiation.cc:407] Connection header received
0809 22:38:20.963593 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:20.963606 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:20.964036 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:20.964279 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:38:20.965400 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.967350 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.972216 (+ 4866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.974438 (+ 2222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.974647 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.974904 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:20.975213 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 22:38:20.975408 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2463,"thread_start_us":2180,"threads_started":1}
I20260809 22:38:20.977373 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.961214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:40279 (local address 127.0.0.1:45750)
0809 22:38:20.964046 (+ 2832us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:20.964066 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:20.964198 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:20.964935 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:20.964941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:20.964975 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:38:20.965491 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.965507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.967631 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.967637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.970758 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:20.970783 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.976450 (+ 5667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.976455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:20.976665 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.976691 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 22:38:20.976800 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 22:38:20.976943 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2684,"mutex_wait_us":36,"thread_start_us":3036,"threads_started":1}
I20260809 22:38:20.977557 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:20.961909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.28.243.67:40279)
0809 22:38:20.963478 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:20.963487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:20.964327 (+ 840us) server_negotiation.cc:407] Connection header received
0809 22:38:20.964435 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:20.964443 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:20.964701 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:20.964931 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:38:20.965705 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.967407 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.972325 (+ 4918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:20.976140 (+ 3815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:20.976313 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:20.976677 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:20.976959 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 22:38:20.977196 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1295,"thread_start_us":1026,"threads_started":1}
I20260809 22:38:20.977303 8600 tablet_service.cc:1467] Processing CreateTablet for tablet fd1a1b770db940aa90f01a693be0dc03 (DEFAULT_TABLE table=test-table [id=cb8c47fd05e94e60ae5e35096734d6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:38:20.979267 8600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1a1b770db940aa90f01a693be0dc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:38:20.980297 8460 tablet_service.cc:1467] Processing CreateTablet for tablet fd1a1b770db940aa90f01a693be0dc03 (DEFAULT_TABLE table=test-table [id=cb8c47fd05e94e60ae5e35096734d6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:38:20.982072 8460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1a1b770db940aa90f01a693be0dc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:38:20.984440 8743 tablet_service.cc:1467] Processing CreateTablet for tablet fd1a1b770db940aa90f01a693be0dc03 (DEFAULT_TABLE table=test-table [id=cb8c47fd05e94e60ae5e35096734d6e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:38:20.986382 8743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd1a1b770db940aa90f01a693be0dc03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:38:21.000275 8834 tablet_bootstrap.cc:492] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Bootstrap starting.
I20260809 22:38:21.003674 8836 tablet_bootstrap.cc:492] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Bootstrap starting.
I20260809 22:38:21.006470 8834 tablet_bootstrap.cc:654] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Neither blocks nor log segments found. Creating new log.
I20260809 22:38:21.007273 8835 tablet_bootstrap.cc:492] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Bootstrap starting.
I20260809 22:38:21.008122 8834 log.cc:824] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Log is configured to *not* fsync() on all Append() calls
I20260809 22:38:21.008458 8836 tablet_bootstrap.cc:654] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Neither blocks nor log segments found. Creating new log.
I20260809 22:38:21.010076 8836 log.cc:824] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:38:21.012022 8835 tablet_bootstrap.cc:654] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Neither blocks nor log segments found. Creating new log.
I20260809 22:38:21.013479 8835 log.cc:824] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Log is configured to *not* fsync() on all Append() calls
I20260809 22:38:21.013932 8836 tablet_bootstrap.cc:492] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: No bootstrap required, opened a new log
I20260809 22:38:21.014039 8834 tablet_bootstrap.cc:492] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: No bootstrap required, opened a new log
I20260809 22:38:21.014253 8836 ts_tablet_manager.cc:1397] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260809 22:38:21.014305 8834 ts_tablet_manager.cc:1397] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260809 22:38:21.023300 8835 tablet_bootstrap.cc:492] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: No bootstrap required, opened a new log
I20260809 22:38:21.023753 8835 ts_tablet_manager.cc:1397] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260809 22:38:21.025965 8836 raft_consensus.cc:348] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.025965 8834 raft_consensus.cc:348] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.026479 8834 raft_consensus.cc:374] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:38:21.026481 8836 raft_consensus.cc:374] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:38:21.026592 8836 raft_consensus.cc:729] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d0cc902a79475b874d9f5b9a35a56d, State: Initialized, Role: FOLLOWER
I20260809 22:38:21.026592 8834 raft_consensus.cc:729] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e48f8c5e4b433da076490151da0136, State: Initialized, Role: FOLLOWER
I20260809 22:38:21.027303 8836 consensus_queue.cc:260] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.027295 8834 consensus_queue.cc:260] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.029404 8834 ts_tablet_manager.cc:1428] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260809 22:38:21.031272 8812 heartbeater.cc:503] Master 127.28.243.126:33157 was elected leader, sending a full tablet report...
I20260809 22:38:21.032752 8836 ts_tablet_manager.cc:1428] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260809 22:38:21.035251 8835 raft_consensus.cc:348] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.035784 8835 raft_consensus.cc:374] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:38:21.035912 8835 raft_consensus.cc:729] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb942da9e1cc448f855cf709efd10406, State: Initialized, Role: FOLLOWER
I20260809 22:38:21.036649 8835 consensus_queue.cc:260] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.037945 8671 heartbeater.cc:503] Master 127.28.243.126:33157 was elected leader, sending a full tablet report...
I20260809 22:38:21.038684 8835 ts_tablet_manager.cc:1428] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
W20260809 22:38:21.116741 8813 tablet.cc:2369] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:38:21.145349 8529 tablet.cc:2369] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:38:21.214586 8842 raft_consensus.cc:482] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:38:21.214982 8842 raft_consensus.cc:504] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.216969 8842 leader_election.cc:288] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8e48f8c5e4b433da076490151da0136 (127.28.243.65:43251), a1d0cc902a79475b874d9f5b9a35a56d (127.28.243.67:40279)
I20260809 22:38:21.224447 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.217618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:49773 (local address 127.28.243.67:40279)
0809 22:38:21.217832 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:21.217838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:21.218394 (+ 556us) server_negotiation.cc:407] Connection header received
0809 22:38:21.218641 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:21.218645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:21.218721 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:21.218820 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:38:21.219488 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.220322 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.222498 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.223340 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.223545 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.223643 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:21.224010 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 22:38:21.224156 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 22:38:21.224457 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.217497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:54275 (local address 127.28.243.65:43251)
0809 22:38:21.217740 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:21.217747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:21.218470 (+ 723us) server_negotiation.cc:407] Connection header received
0809 22:38:21.218687 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:21.218691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:21.218755 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:21.218874 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:38:21.219669 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.220432 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.222457 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.223346 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.223551 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.223649 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:21.224026 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 22:38:21.224171 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 22:38:21.224658 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.217337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:40279 (local address 127.28.243.66:49773)
0809 22:38:21.218246 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:21.218271 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:21.218483 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:21.218870 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:21.218875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:21.218912 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:38:21.219319 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:21.219334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.220467 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.220475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:21.222287 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:21.222299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.223609 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.223617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:21.223802 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.223863 (+ 61us) client_negotiation.cc:769] Sending connection context
0809 22:38:21.223980 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 22:38:21.224137 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":26,"thread_start_us":467,"threads_started":1}
I20260809 22:38:21.225132 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.217362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:43251 (local address 127.28.243.66:54275)
0809 22:38:21.218345 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0809 22:38:21.218361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:38:21.218531 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:38:21.218887 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:38:21.218892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:38:21.218987 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:38:21.219485 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:21.219496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.220625 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.220632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:21.222283 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:38:21.222297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.223649 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.223657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:38:21.223842 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.223884 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 22:38:21.223997 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 22:38:21.224145 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":15,"spinlock_wait_cycles":28416,"thread_start_us":650,"threads_started":1}
I20260809 22:38:21.225919 8764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1a1b770db940aa90f01a693be0dc03" candidate_uuid: "cb942da9e1cc448f855cf709efd10406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" is_pre_election: true
I20260809 22:38:21.226138 8482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1a1b770db940aa90f01a693be0dc03" candidate_uuid: "cb942da9e1cc448f855cf709efd10406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e48f8c5e4b433da076490151da0136" is_pre_election: true
I20260809 22:38:21.226498 8764 raft_consensus.cc:2454] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb942da9e1cc448f855cf709efd10406 in term 0.
I20260809 22:38:21.226686 8482 raft_consensus.cc:2454] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb942da9e1cc448f855cf709efd10406 in term 0.
I20260809 22:38:21.227269 8551 leader_election.cc:302] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d0cc902a79475b874d9f5b9a35a56d, cb942da9e1cc448f855cf709efd10406; no voters:
I20260809 22:38:21.227710 8842 raft_consensus.cc:2781] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:38:21.227864 8842 raft_consensus.cc:482] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:38:21.227981 8842 raft_consensus.cc:3037] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:38:21.230342 8842 raft_consensus.cc:504] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.230958 8842 leader_election.cc:288] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [CANDIDATE]: Term 1 election: Requested vote from peers b8e48f8c5e4b433da076490151da0136 (127.28.243.65:43251), a1d0cc902a79475b874d9f5b9a35a56d (127.28.243.67:40279)
I20260809 22:38:21.231617 8764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1a1b770db940aa90f01a693be0dc03" candidate_uuid: "cb942da9e1cc448f855cf709efd10406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d0cc902a79475b874d9f5b9a35a56d"
I20260809 22:38:21.231617 8482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd1a1b770db940aa90f01a693be0dc03" candidate_uuid: "cb942da9e1cc448f855cf709efd10406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e48f8c5e4b433da076490151da0136"
I20260809 22:38:21.231855 8764 raft_consensus.cc:3037] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:38:21.231866 8482 raft_consensus.cc:3037] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:38:21.235239 8482 raft_consensus.cc:2454] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb942da9e1cc448f855cf709efd10406 in term 1.
I20260809 22:38:21.235239 8764 raft_consensus.cc:2454] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb942da9e1cc448f855cf709efd10406 in term 1.
I20260809 22:38:21.235769 8551 leader_election.cc:302] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1d0cc902a79475b874d9f5b9a35a56d, cb942da9e1cc448f855cf709efd10406; no voters:
I20260809 22:38:21.236129 8842 raft_consensus.cc:2781] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:38:21.236650 8842 raft_consensus.cc:686] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 1 LEADER]: Becoming Leader. State: Replica: cb942da9e1cc448f855cf709efd10406, State: Running, Role: LEADER
I20260809 22:38:21.237229 8842 consensus_queue.cc:237] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } }
I20260809 22:38:21.242211 8333 catalog_manager.cc:5495] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb942da9e1cc448f855cf709efd10406 (127.28.243.66). New cstate: current_term: 1 leader_uuid: "cb942da9e1cc448f855cf709efd10406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb942da9e1cc448f855cf709efd10406" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 } health_report { overall_health: UNKNOWN } } }
W20260809 22:38:21.256403 8672 tablet.cc:2369] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:38:21.277441 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.270341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.28.243.66:46559)
0809 22:38:21.270574 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:21.270579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:21.270601 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:38:21.270704 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:21.270708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:21.270797 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:21.270982 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:38:21.271487 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.272314 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.274414 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.275230 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.275344 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.276078 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:38:21.276813 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:38:21.276941 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:21.277059 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 22:38:21.277135 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 22:38:21.286083 8482 raft_consensus.cc:1261] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 1 FOLLOWER]: Refusing update from remote peer cb942da9e1cc448f855cf709efd10406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:38:21.286084 8764 raft_consensus.cc:1261] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 1 FOLLOWER]: Refusing update from remote peer cb942da9e1cc448f855cf709efd10406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:38:21.287362 8842 consensus_queue.cc:1035] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e48f8c5e4b433da076490151da0136" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 43251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:38:21.287752 8846 consensus_queue.cc:1035] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d0cc902a79475b874d9f5b9a35a56d" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:38:21.302335 8851 mvcc.cc:205] Tried to move back new op lower bound from 7316746654857080832 to 7316746654672310272. Current Snapshot: MvccSnapshot[applied={T|T < 7316746654857080832}]
I20260809 22:38:21.316325 8332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:38:21.316732 8332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:38:21.320731 8332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0ed3385ba3744e491199902bef8d15d: Sending DeleteTablet for 3 replicas of tablet fd1a1b770db940aa90f01a693be0dc03
I20260809 22:38:21.322108 8460 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1a1b770db940aa90f01a693be0dc03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:38:21 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:50612
I20260809 22:38:21.322126 8600 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1a1b770db940aa90f01a693be0dc03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:38:21 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:56188
I20260809 22:38:21.322243 8743 tablet_service.cc:1514] Processing DeleteTablet for tablet fd1a1b770db940aa90f01a693be0dc03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:38:21 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:45750
I20260809 22:38:21.324891 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 8396
I20260809 22:38:21.333364 8859 tablet_replica.cc:330] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: stopping tablet replica
I20260809 22:38:21.334064 8859 raft_consensus.cc:2229] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:38:21.334812 8859 raft_consensus.cc:2258] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:38:21.337272 8860 tablet_replica.cc:330] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: stopping tablet replica
I20260809 22:38:21.337531 8861 tablet_replica.cc:330] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: stopping tablet replica
I20260809 22:38:21.337779 8860 raft_consensus.cc:2229] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:38:21.337965 8861 raft_consensus.cc:2229] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:38:21.338534 8861 raft_consensus.cc:2258] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:38:21.338567 8860 raft_consensus.cc:2258] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:38:21.341265 8860 ts_tablet_manager.cc:1905] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:38:21.341899 8861 ts_tablet_manager.cc:1905] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:38:21.348243 8859 ts_tablet_manager.cc:1905] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:38:21.349735 8861 ts_tablet_manager.cc:1918] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:38:21.349956 8861 log.cc:1196] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/fd1a1b770db940aa90f01a693be0dc03
I20260809 22:38:21.350028 8860 ts_tablet_manager.cc:1918] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:38:21.350222 8860 log.cc:1196] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/fd1a1b770db940aa90f01a693be0dc03
I20260809 22:38:21.350621 8861 ts_tablet_manager.cc:1939] T fd1a1b770db940aa90f01a693be0dc03 P a1d0cc902a79475b874d9f5b9a35a56d: Deleting consensus metadata
I20260809 22:38:21.350837 8860 ts_tablet_manager.cc:1939] T fd1a1b770db940aa90f01a693be0dc03 P cb942da9e1cc448f855cf709efd10406: Deleting consensus metadata
I20260809 22:38:21.352763 8316 catalog_manager.cc:4841] TS a1d0cc902a79475b874d9f5b9a35a56d (127.28.243.67:40279): tablet fd1a1b770db940aa90f01a693be0dc03 (table test-table [id=cb8c47fd05e94e60ae5e35096734d6e7]) successfully deleted
I20260809 22:38:21.353155 8316 catalog_manager.cc:4841] TS cb942da9e1cc448f855cf709efd10406 (127.28.243.66:46559): tablet fd1a1b770db940aa90f01a693be0dc03 (table test-table [id=cb8c47fd05e94e60ae5e35096734d6e7]) successfully deleted
Aug 09 22:38:21 dist-test-slave-rgb1 krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:38:21.357842 8859 ts_tablet_manager.cc:1918] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:38:21.358047 8859 log.cc:1196] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/fd1a1b770db940aa90f01a693be0dc03
I20260809 22:38:21.358814 8859 ts_tablet_manager.cc:1939] T fd1a1b770db940aa90f01a693be0dc03 P b8e48f8c5e4b433da076490151da0136: Deleting consensus metadata
I20260809 22:38:21.360966 8318 catalog_manager.cc:4841] TS b8e48f8c5e4b433da076490151da0136 (127.28.243.65:43251): tablet fd1a1b770db940aa90f01a693be0dc03 (table test-table [id=cb8c47fd05e94e60ae5e35096734d6e7]) successfully deleted
I20260809 22:38:21.365239 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.326613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.28.243.65:43251)
0809 22:38:21.326811 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:21.326817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:21.339810 (+ 12993us) server_negotiation.cc:407] Connection header received
0809 22:38:21.340331 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:21.340337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:21.340426 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:21.340523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:21.342628 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.343779 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.345838 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.346530 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.346636 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.356809 (+ 10173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:21.356826 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:21.356846 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:21.356896 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:21.361790 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:21.362616 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:21.362625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:21.362629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:21.362802 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:21.363373 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:21.363379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:21.363382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:21.363893 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:21.364242 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:21.364551 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 22:38:21.364833 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:38:21.365964 8523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:21.462667 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 8396
I20260809 22:38:21.491470 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 8534
Aug 09 22:38:21 dist-test-slave-rgb1 krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:38:21.510207 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.492642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.28.243.66:46559)
0809 22:38:21.492823 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:21.492828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:21.492852 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:38:21.492910 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:21.492914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:21.492996 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:21.493084 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:21.494878 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.495901 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.497848 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.498408 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.498553 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.503109 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:21.503125 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:21.503141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:21.503183 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:21.506831 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:21.507730 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:21.507737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:21.507740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:21.507901 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:21.508370 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:21.508375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:21.508377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:21.508856 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:21.509221 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:21.509519 (+ 298us) server_negotiation.cc:299] Negotiation successful
0809 22:38:21.509787 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 22:38:21.511209 8666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:21.632103 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 8534
I20260809 22:38:21.653898 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 8677
Aug 09 22:38:21 dist-test-slave-rgb1 krb5kdc[8287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:38:21.670099 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:38:21.654734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.28.243.67:40279)
0809 22:38:21.654927 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 22:38:21.654936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:38:21.654963 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:38:21.655245 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:38:21.655251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:38:21.655322 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:38:21.655443 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:38:21.656783 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.657511 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.658703 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:38:21.659105 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:38:21.659240 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:38:21.663962 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:38:21.663979 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:38:21.663996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:38:21.664034 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:38:21.666938 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:21.667659 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:21.667668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:21.667671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:21.667836 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:38:21.668238 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:38:21.668243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:38:21.668247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:38:21.668838 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:38:21.669153 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 22:38:21.669501 (+ 348us) server_negotiation.cc:299] Negotiation successful
0809 22:38:21.669742 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 22:38:21.670818 8805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:21.762327 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 8677
I20260809 22:38:21.782130 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 8300
I20260809 22:38:21.783308 8363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:38:21.899591 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 8300
2026-08-09T22:38:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5093 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:38:22 dist-test-slave-rgb1 krb5kdc[8869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:38:22 dist-test-slave-rgb1 krb5kdc[8869](info): set up 2 sockets
Aug 09 22:38:22 dist-test-slave-rgb1 krb5kdc[8869](info): commencing operation
krb5kdc: starting...
W20260809 22:38:23.111645 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:38:23 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:38:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/rangeradmin_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/rangeradmin_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/rangerlookup_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/rangerlookup_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
I20260809 22:38:23.589424 29645 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmuNQXR/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/postgres -l logfile start
2026-08-09 22:38:26.813 UTC [8902] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 22:38:26.813 UTC [8902] LOG: listening on IPv4 address "127.28.243.84", port 58203
2026-08-09 22:38:26.818 UTC [8902] LOG: listening on Unix socket "/tmp/.s.PGSQL.58203"
2026-08-09 22:38:26.842 UTC [8904] LOG: database system was shut down at 2026-08-09 22:38:24 UTC
2026-08-09 22:38:26.846 UTC [8902] LOG: database system is ready to accept connections
I20260809 22:38:27.917713 29645 mini_postgres.cc:96] Postgres bound to 58203
2026-08-09 22:38:27.943 UTC [8914] FATAL: database "slave" does not exist
127.28.243.84:58203 - accepting connections
I20260809 22:38:27.944880 29645 mini_ranger.cc:161] Starting Ranger...
I20260809 22:38:27.981467 29645 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 22:38:28.656538 29645 mini_ranger.cc:87] Created ranger Postgres database
I20260809 22:38:28.656724 29645 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-admin
2026-08-09 22:38:28,998 [I] DB FLAVOR :POSTGRES
2026-08-09 22:38:28,999 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 22:38:28,999 [I] Checking connection..
2026-08-09 22:38:28,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 22:38:29,268 [I] Checking connection passed.
2026-08-09 22:38:29,268 [I] --------- Verifying version history table ---------
2026-08-09 22:38:29,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 22:38:29,550 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 22:38:29,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 22:38:29,818 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 22:38:29,818 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 22:38:29,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 22:38:30.104 UTC [8982] WARNING: there is no transaction in progress
2026-08-09 22:38:30,112 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 22:38:30,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 22:38:30,369 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 22:38:30,369 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 22:38:30,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 22:38:30,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 22:38:30,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rgb1.c.gcp-upstream.internal','N') ;"
2026-08-09 22:38:31,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 22:38:31,512 [I] Table x_portal_user does not exist in database ranger
2026-08-09 22:38:31,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 22:38:31,779 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 22:38:31,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 22:38:32,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 22:38:32,307 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 22:38:32,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 22:38:33.529 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.549 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.569 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.591 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.616 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.640 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.706 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.714 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.731 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.744 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.757 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.775 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.787 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.795 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.804 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:33.813 UTC [9098] WARNING: there is no transaction in progress
2026-08-09 22:38:34,713 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 22:38:34,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 22:38:35,010 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 22:38:35,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 22:38:35,270 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 22:38:35,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 22:38:35,552 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 22:38:35,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 22:38:35,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 22:38:36,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rgb1.c.gcp-upstream.internal';"
2026-08-09 22:38:36,346 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 22:38:36,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 22:38:36,598 [I] DB_PATCHES have already been applied
I20260809 22:38:36.604679 29645 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmuNQXR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 22:38:36.604781 29645 mini_ranger.cc:193] Using host: 127.28.243.84
I20260809 22:38:36.627214 29645 mini_ranger.cc:227] Ranger admin URL: http://127.28.243.84:41159
Aug 09, 2026 10:38:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 10:38:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 10:38:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 10:38:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 10:38:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 10:38:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 10:38:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 22:38:37.889667 29645 mini_ranger.cc:160] Time spent starting Ranger: real 9.945s user 0.006s sys 0.068s
Aug 09, 2026 10:38:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 10:38:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 10:38:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 10:38:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 10:39:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 22:39:00.946512 29645 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/rangerkms_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/rangerkms_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.243.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 22:39:01 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315141, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 22:39:01.193843 29645 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 22:39:01.225844 29645 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 22:39:02.454353 29645 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 22:39:02.454543 29645 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms
I20260809 22:39:02.454674 29645 mini_ranger_kms.cc:227] Using postgres at 127.28.243.84:58203
2026-08-09 22:39:03,081 [I] DB FLAVOR :POSTGRES
2026-08-09 22:39:03,082 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 22:39:03,082 [I] Checking connection
2026-08-09 22:39:03,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 22:39:03,371 [I] connection success
2026-08-09 22:39:03,371 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 22:39:03,371 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 22:39:03,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 22:39:03,625 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 22:39:03,625 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 22:39:03,626 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 22:39:03,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:58203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 22:39:03,900 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 22:39:04.442924 29645 mini_ranger_kms.cc:310] Created kms service
I20260809 22:39:04.635349 29645 mini_ranger_kms.cc:326] Created kudu user
I20260809 22:39:04.687505 29645 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 22:39:04 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315141, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:39:05.158980 29645 mini_ranger_kms.cc:384] Added ranger policy
I20260809 22:39:05.160606 29645 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 22:39:05.160694 29645 mini_ranger_kms.cc:240] Using host: 127.28.243.84
I20260809 22:39:05.193921 29645 mini_ranger_kms.cc:276] Ranger KMS PID: 9289
I20260809 22:39:05.194296 29645 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.243.84:39629
Aug 09, 2026 10:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 10:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 10:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 10:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 10:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 10:39:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 22:39:07 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786315147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:39:07.639804 29645 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.446s user 0.147s sys 0.512s
I20260809 22:39:07.640211 29645 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 22:39:07.640466 29645 mini_ranger_kms.cc:401] 127.28.243.84:39629/kms/v1/keys
Aug 09 22:39:08 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.84@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
Aug 09 22:39:08 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.84@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
Aug 09 22:39:10 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:39:10.978093 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39791
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39791
--ranger_config_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:39629/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:39:11.367589 9342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:39:11.367911 9342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:39:11.368067 9342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:39:11.376438 9342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:39:11.376624 9342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:39:11.376740 9342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:39:11.376859 9342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:39:11.376951 9342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:39:11.387687 9342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.243.84:39629/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39791
--ranger_config_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:39:11.389194 9342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:39:11.390522 9342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:39:11.407680 9347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:11.407717 9348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:11.408810 9350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:39:11.409498 9342 server_base.cc:1034] running on GCE node
I20260809 22:39:11.410396 9342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:39:11.412035 9342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:39:11.413300 9342 hybrid_clock.cc:648] HybridClock initialized: now 1786315151413242 us; error 45 us; skew 500 ppm
I20260809 22:39:11.413699 9342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:39:11 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:39:11.421957 9342 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:39:11.423902 9342 webserver.cc:460] Webserver started at http://127.28.243.126:35009/ using document root <none> and password file <none>
I20260809 22:39:11.424716 9342 fs_manager.cc:359] Metadata directory not provided
I20260809 22:39:11.424870 9342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:39:11.425213 9342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:39:11 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315151, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:39:11.488698 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "2f03be66b469461ca33d4c8c00944f23"
format_stamp: "Formatted at 2026-08-09 22:39:11 on dist-test-slave-rgb1"
server_key: "d86ba1558b8ed0ec34ede34925e6f5e0"
server_key_iv: "56938fd18b71bc50f3acd59d356bb59b"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:11.490020 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "2f03be66b469461ca33d4c8c00944f23"
format_stamp: "Formatted at 2026-08-09 22:39:11 on dist-test-slave-rgb1"
server_key: "d86ba1558b8ed0ec34ede34925e6f5e0"
server_key_iv: "56938fd18b71bc50f3acd59d356bb59b"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:11.497866 9342 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260809 22:39:11.516834 9358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:11.518879 9342 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:39:11.519142 9342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "2f03be66b469461ca33d4c8c00944f23"
format_stamp: "Formatted at 2026-08-09 22:39:11 on dist-test-slave-rgb1"
server_key: "d86ba1558b8ed0ec34ede34925e6f5e0"
server_key_iv: "56938fd18b71bc50f3acd59d356bb59b"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:11.519421 9342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:39:11.538801 9342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:39:11.543489 9342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:39:11.543845 9342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:39:11.567191 9342 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:39791
I20260809 22:39:11.567219 9410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:39791 every 8 connection(s)
I20260809 22:39:11.568988 9342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:39:11.573561 9411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:39:11.579159 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 9342
I20260809 22:39:11.579386 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:39:11.580229 29645 external_mini_cluster.cc:1428] Setting key f2418b7fa1a4fac61ec7c9630fccdfca
I20260809 22:39:11.588161 9411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23: Bootstrap starting.
I20260809 22:39:11.593784 9411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23: Neither blocks nor log segments found. Creating new log.
I20260809 22:39:11.595748 9411 log.cc:824] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23: Log is configured to *not* fsync() on all Append() calls
I20260809 22:39:11.601050 9411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23: No bootstrap required, opened a new log
I20260809 22:39:11.611271 9411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f03be66b469461ca33d4c8c00944f23" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39791 } }
I20260809 22:39:11.611729 9411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:39:11.611897 9411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f03be66b469461ca33d4c8c00944f23, State: Initialized, Role: FOLLOWER
I20260809 22:39:11.613075 9411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f03be66b469461ca33d4c8c00944f23" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39791 } }
I20260809 22:39:11.613399 9411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:39:11.613633 9411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:39:11.613873 9411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 0 FOLLOWER]: Advancing to term 1
Aug 09 22:39:11 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:39:11.617251 9411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f03be66b469461ca33d4c8c00944f23" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39791 } }
I20260809 22:39:11.617928 9411 leader_election.cc:302] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f03be66b469461ca33d4c8c00944f23; no voters:
I20260809 22:39:11.619166 9411 leader_election.cc:288] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:39:11.619596 9416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:39:11.623864 9416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [term 1 LEADER]: Becoming Leader. State: Replica: 2f03be66b469461ca33d4c8c00944f23, State: Running, Role: LEADER
I20260809 22:39:11.624720 9416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f03be66b469461ca33d4c8c00944f23" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39791 } }
I20260809 22:39:11.628805 9411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:39:11.629336 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:11.582678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.28.243.126:39791)
0809 22:39:11.598321 (+ 15643us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:11.598333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:11.598385 (+ 52us) server_negotiation.cc:407] Connection header received
0809 22:39:11.601148 (+ 2763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:11.601180 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:11.601715 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:11.602209 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:11.604100 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:11.606277 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:11.608227 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:11.609460 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:11.609687 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:11.615609 (+ 5922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:11.615627 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:11.615650 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:11.615717 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:11.621052 (+ 5335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:11.622547 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:11.622562 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:11.622577 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:11.622862 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:11.623479 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:11.623488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:11.623494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:11.624659 (+ 1165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:11.626918 (+ 2259us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:11.627280 (+ 362us) server_negotiation.cc:299] Negotiation successful
0809 22:39:11.627748 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15402,"thread_start_us":15126,"threads_started":1}
I20260809 22:39:11.631594 9417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f03be66b469461ca33d4c8c00944f23. Latest consensus state: current_term: 1 leader_uuid: "2f03be66b469461ca33d4c8c00944f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f03be66b469461ca33d4c8c00944f23" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39791 } } }
I20260809 22:39:11.631865 9418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f03be66b469461ca33d4c8c00944f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f03be66b469461ca33d4c8c00944f23" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39791 } } }
I20260809 22:39:11.632241 9418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [sys.catalog]: This master's current role is: LEADER
I20260809 22:39:11.632249 9417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23 [sys.catalog]: This master's current role is: LEADER
I20260809 22:39:11.644364 9421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:39:11.651811 9421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:39:11.666359 9411 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 22:39:11.667760 9421 catalog_manager.cc:1347] Generated new cluster ID: abc897cb85044ea9bdab9f251bc76cfa
I20260809 22:39:11.667881 9421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:39:11.706561 9421 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:39:11.709610 9421 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:39:11.720012 9421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23: Generated new TSK 0
I20260809 22:39:11.721706 9421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:39:12.645614 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:39629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39791
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:39:12 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786315152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 22:39:13.340945 9459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:39:13.341487 9459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:39:13.341766 9459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:39:13.354991 9459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:39:13.355273 9459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:39:13.355530 9459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:39:13.370088 9459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.243.84:39629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:39:13.371820 9459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:39:13.373948 9459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:39:13.409273 9470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:13.410424 9471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:13.428835 9473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:39:13.429801 9459 server_base.cc:1034] running on GCE node
I20260809 22:39:13.430852 9459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:39:13.432816 9459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:39:13.434212 9459 hybrid_clock.cc:648] HybridClock initialized: now 1786315153434041 us; error 111 us; skew 500 ppm
I20260809 22:39:13.434782 9459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:39:13 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:39:13.440887 9459 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:39:13.444453 9459 webserver.cc:460] Webserver started at http://127.28.243.65:41645/ using document root <none> and password file <none>
I20260809 22:39:13.445914 9459 fs_manager.cc:359] Metadata directory not provided
I20260809 22:39:13.446179 9459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:39:13.446774 9459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:39:13 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:39:13.491547 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "0fde9703ec6a44cfb0181e07738a7c25"
format_stamp: "Formatted at 2026-08-09 22:39:13 on dist-test-slave-rgb1"
server_key: "bd58aee2a8b8d1180e4ff980e2ac2b7c"
server_key_iv: "a12a11806855811d73a0d892f065107c"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:13.493652 9459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "0fde9703ec6a44cfb0181e07738a7c25"
format_stamp: "Formatted at 2026-08-09 22:39:13 on dist-test-slave-rgb1"
server_key: "bd58aee2a8b8d1180e4ff980e2ac2b7c"
server_key_iv: "a12a11806855811d73a0d892f065107c"
server_key_version: "kuduclusterkey@0"
Aug 09 22:39:13 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:39:13.513177 9459 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.016s sys 0.000s
I20260809 22:39:13.543387 9481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:13.545990 9459 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260809 22:39:13.546306 9459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "0fde9703ec6a44cfb0181e07738a7c25"
format_stamp: "Formatted at 2026-08-09 22:39:13 on dist-test-slave-rgb1"
server_key: "bd58aee2a8b8d1180e4ff980e2ac2b7c"
server_key_iv: "a12a11806855811d73a0d892f065107c"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:13.546581 9459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:39:13.580379 9459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:39:13.587776 9459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:39:13.588428 9459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:39:13.590533 9459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:39:13.593582 9459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:39:13.593777 9459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:13.594072 9459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:39:13.594247 9459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:13.710753 9459 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:34559
I20260809 22:39:13.710903 9598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:34559 every 8 connection(s)
I20260809 22:39:13.712854 9459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:39:13.722127 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 9459
I20260809 22:39:13.722375 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:39:13.723222 29645 external_mini_cluster.cc:1428] Setting key 977284c88292fb322465d3aac8860156
Aug 09 22:39:13 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315153, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:39:13.755295 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:13.718766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:55773 (local address 127.28.243.126:39791)
0809 22:39:13.725091 (+ 6325us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:13.725098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:13.727877 (+ 2779us) server_negotiation.cc:407] Connection header received
0809 22:39:13.731378 (+ 3501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:13.731387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:13.731487 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:13.731617 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:13.734199 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:13.735520 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:13.738676 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:13.739359 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:13.739475 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:13.744677 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:13.744684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:13.744689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:13.744747 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:13.749845 (+ 5098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:13.751191 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:13.751198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:13.751201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:13.751478 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:13.752244 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:13.752249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:13.752252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:13.752768 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:13.752918 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:13.754353 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0809 22:39:13.754662 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6189,"thread_start_us":7047,"threads_started":1}
I20260809 22:39:13.758459 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:13.717916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39791 (local address 127.28.243.65:55773)
0809 22:39:13.725697 (+ 7781us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:13.727083 (+ 1386us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:13.730745 (+ 3662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:13.732131 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:13.732145 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:13.732945 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:39:13.733949 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:13.733978 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:13.735798 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:13.735804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:13.738465 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:13.738482 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:13.739903 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:13.739913 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:13.740290 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:13.741262 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:39:13.741298 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:39:13.744313 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:39:13.750136 (+ 5823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:39:13.750147 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:39:13.750169 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:39:13.751016 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:39:13.751658 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:39:13.751663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:39:13.751667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:39:13.752102 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:39:13.753020 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:39:13.753030 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:39:13.753686 (+ 656us) client_negotiation.cc:769] Sending connection context
0809 22:39:13.755914 (+ 2228us) client_negotiation.cc:240] Negotiation successful
0809 22:39:13.756660 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7414,"thread_start_us":7844,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:39:13.761018 9599 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39791
I20260809 22:39:13.761415 9599 heartbeater.cc:464] Registering TS with master...
I20260809 22:39:13.762645 9599 heartbeater.cc:511] Master 127.28.243.126:39791 requested a full tablet report, sending...
I20260809 22:39:13.766024 9375 ts_manager.cc:194] Registered new tserver with Master: 0fde9703ec6a44cfb0181e07738a7c25 (127.28.243.65:34559)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
I20260809 22:39:13.768203 9375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:55773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:39:13.809180 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:39629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39791
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:39:13 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315152, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
W20260809 22:39:14.466445 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:39:14.466910 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:39:14.467165 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:39:14.477957 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:39:14.478223 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:39:14.478468 9605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:39:14.494315 9605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.243.84:39629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:39:14.495777 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:39:14.497543 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:39:14.520372 9619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:14.520772 9620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:14.521621 9622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:39:14.523365 9605 server_base.cc:1034] running on GCE node
I20260809 22:39:14.524225 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:39:14.525759 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:39:14.527029 9605 hybrid_clock.cc:648] HybridClock initialized: now 1786315154526950 us; error 69 us; skew 500 ppm
I20260809 22:39:14.527411 9605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:39:14 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:39:14.533252 9605 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:39:14.536500 9605 webserver.cc:460] Webserver started at http://127.28.243.66:43631/ using document root <none> and password file <none>
I20260809 22:39:14.537755 9605 fs_manager.cc:359] Metadata directory not provided
I20260809 22:39:14.537910 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:39:14.538280 9605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:39:14 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:39:14.559427 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "749cc4faf4cc4cfaa674aac3acab20fe"
format_stamp: "Formatted at 2026-08-09 22:39:14 on dist-test-slave-rgb1"
server_key: "1798d689073550d56a4795c187211081"
server_key_iv: "b152191043e2a50a389ea99c4cfd90c4"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:14.560784 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "749cc4faf4cc4cfaa674aac3acab20fe"
format_stamp: "Formatted at 2026-08-09 22:39:14 on dist-test-slave-rgb1"
server_key: "1798d689073550d56a4795c187211081"
server_key_iv: "b152191043e2a50a389ea99c4cfd90c4"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:14.570058 9605 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.001s
I20260809 22:39:14.601235 9629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:14.603718 9605 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.007s
I20260809 22:39:14.604161 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "749cc4faf4cc4cfaa674aac3acab20fe"
format_stamp: "Formatted at 2026-08-09 22:39:14 on dist-test-slave-rgb1"
server_key: "1798d689073550d56a4795c187211081"
server_key_iv: "b152191043e2a50a389ea99c4cfd90c4"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:14.604789 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:39:14.656847 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:39:14.665823 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:39:14.666508 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:39:14.680737 9605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:39:14.694181 9605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:39:14.694454 9605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.001s
I20260809 22:39:14.694762 9605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:39:14.694927 9605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:14.773548 9599 heartbeater.cc:503] Master 127.28.243.126:39791 was elected leader, sending a full tablet report...
I20260809 22:39:15.141848 9605 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:38019
I20260809 22:39:15.142100 9745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:38019 every 8 connection(s)
I20260809 22:39:15.144213 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:39:15.149211 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 9605
I20260809 22:39:15.149466 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:39:15.150287 29645 external_mini_cluster.cc:1428] Setting key 3db2fca32d1f7aff406dbfebad0b3aab
Aug 09 22:39:15 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315154, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:39:15.221254 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:15.151410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:50661 (local address 127.28.243.126:39791)
0809 22:39:15.190535 (+ 39125us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:15.190543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:15.190569 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:39:15.190653 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:15.190658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:15.190753 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:15.190873 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:15.199015 (+ 8142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:15.200588 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:15.203537 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:15.204343 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:15.204460 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:15.210717 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:15.210724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:15.210730 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:15.210797 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:15.216287 (+ 5490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:15.217580 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:15.217587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:15.217590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:15.217847 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:15.218549 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:15.218554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:15.218558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:15.218984 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:15.219163 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:15.220366 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0809 22:39:15.220663 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38899,"thread_start_us":39623,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
I20260809 22:39:15.224550 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:15.153087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39791 (local address 127.28.243.66:50661)
0809 22:39:15.184739 (+ 31652us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:15.184811 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:15.188178 (+ 3367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:15.193722 (+ 5544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:15.193737 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:15.194534 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:39:15.196064 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:15.196095 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:15.200832 (+ 4737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:15.200838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:15.203321 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:15.203339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:15.205058 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:15.205065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:15.205441 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:15.206553 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:39:15.206584 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:39:15.210357 (+ 3773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:39:15.216547 (+ 6190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:39:15.216581 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:39:15.216602 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:39:15.217406 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:39:15.218017 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:39:15.218021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:39:15.218024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:39:15.218400 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:39:15.219311 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:39:15.219319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:39:15.219844 (+ 525us) client_negotiation.cc:769] Sending connection context
0809 22:39:15.221853 (+ 2009us) client_negotiation.cc:240] Negotiation successful
0809 22:39:15.222556 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31136,"thread_start_us":35351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
I20260809 22:39:15.227056 9747 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39791
I20260809 22:39:15.227450 9747 heartbeater.cc:464] Registering TS with master...
I20260809 22:39:15.228545 9747 heartbeater.cc:511] Master 127.28.243.126:39791 requested a full tablet report, sending...
I20260809 22:39:15.230723 9375 ts_manager.cc:194] Registered new tserver with Master: 749cc4faf4cc4cfaa674aac3acab20fe (127.28.243.66:38019)
I20260809 22:39:15.231969 9375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:50661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
W20260809 22:39:15.269284 29645 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.243.67: real 0.114s user 0.000s sys 0.025s
I20260809 22:39:15.270437 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:39629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39791
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:39:15.886114 9753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:39:15.886454 9753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:39:15.886642 9753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:39:15.894672 9753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:39:15.894870 9753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:39:15.895133 9753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:39:15.908548 9753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.243.84:39629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39791
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:39:15.910202 9753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:39:15.911722 9753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:39:15.930295 9759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:15.930312 9758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:39:15.931178 9761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:39:15.933172 9753 server_base.cc:1034] running on GCE node
I20260809 22:39:15.934262 9753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:39:15.935554 9753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:39:15.936890 9753 hybrid_clock.cc:648] HybridClock initialized: now 1786315155936764 us; error 84 us; skew 500 ppm
I20260809 22:39:15.937335 9753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:39:15 dist-test-slave-rgb1 krb5kdc[8869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:39:15.942001 9753 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:39:15.944000 9753 webserver.cc:460] Webserver started at http://127.28.243.67:38845/ using document root <none> and password file <none>
I20260809 22:39:15.945034 9753 fs_manager.cc:359] Metadata directory not provided
I20260809 22:39:15.945268 9753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:39:15.945710 9753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:39:15 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:39:15.963037 9753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "b68afbeda46048cbb5ddbad465a98314"
format_stamp: "Formatted at 2026-08-09 22:39:15 on dist-test-slave-rgb1"
server_key: "6c1143f351118c022d0953bbd1b164a6"
server_key_iv: "e351f4a20d191ca5a44cdbbe256ac228"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:15.964047 9753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "b68afbeda46048cbb5ddbad465a98314"
format_stamp: "Formatted at 2026-08-09 22:39:15 on dist-test-slave-rgb1"
server_key: "6c1143f351118c022d0953bbd1b164a6"
server_key_iv: "e351f4a20d191ca5a44cdbbe256ac228"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:15.970682 9753 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 22:39:15.992383 9768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:15.994563 9753 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260809 22:39:15.994854 9753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "b68afbeda46048cbb5ddbad465a98314"
format_stamp: "Formatted at 2026-08-09 22:39:15 on dist-test-slave-rgb1"
server_key: "6c1143f351118c022d0953bbd1b164a6"
server_key_iv: "e351f4a20d191ca5a44cdbbe256ac228"
server_key_version: "kuduclusterkey@0"
I20260809 22:39:15.995180 9753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:39:16.016454 9753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:39:16.022534 9753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:39:16.023100 9753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:39:16.024665 9753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:39:16.026512 9753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:39:16.026623 9753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:16.026798 9753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:39:16.026889 9753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:39:16.100904 9753 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:34743
I20260809 22:39:16.101078 9886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:34743 every 8 connection(s)
I20260809 22:39:16.102443 9753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:39:16.105890 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 9753
I20260809 22:39:16.106246 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:39:16.107064 29645 external_mini_cluster.cc:1428] Setting key 463b69d97b3ba62807237991fb9b4e8c
Aug 09 22:39:16 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315155, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:39:16.140420 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.107040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39791 (local address 127.28.243.67:33393)
0809 22:39:16.108759 (+ 1719us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:16.108813 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:16.111954 (+ 3141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:16.112794 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:16.112804 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:16.113889 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:39:16.114817 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.114838 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.116728 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.116733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.118651 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.118666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.119903 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.119910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.120264 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.121558 (+ 1294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:39:16.121589 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:39:16.126229 (+ 4640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:39:16.133300 (+ 7071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:39:16.133324 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:39:16.133357 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:39:16.134517 (+ 1160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:39:16.135296 (+ 779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:39:16.135302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:39:16.135306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:39:16.135825 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:39:16.137263 (+ 1438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:39:16.137273 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:39:16.137846 (+ 573us) client_negotiation.cc:769] Sending connection context
0809 22:39:16.138108 (+ 262us) client_negotiation.cc:240] Negotiation successful
0809 22:39:16.138683 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":879,"threads_started":1}
I20260809 22:39:16.145779 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.106184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:33393 (local address 127.28.243.126:39791)
0809 22:39:16.110723 (+ 4539us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.110728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.110778 (+ 50us) server_negotiation.cc:407] Connection header received
0809 22:39:16.112252 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.112257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.112328 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.112450 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:16.115050 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.116459 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.118889 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.119553 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.119674 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.126624 (+ 6950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:16.126633 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:16.126638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:16.126705 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:16.133021 (+ 6316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.134811 (+ 1790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.134817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.134830 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.135100 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.136065 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.136071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.136076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.136647 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:16.137377 (+ 730us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.144935 (+ 7558us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.145240 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4408,"thread_start_us":4103,"threads_started":1}
I20260809 22:39:16.147204 9887 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39791
I20260809 22:39:16.147524 9887 heartbeater.cc:464] Registering TS with master...
I20260809 22:39:16.148365 9887 heartbeater.cc:511] Master 127.28.243.126:39791 requested a full tablet report, sending...
I20260809 22:39:16.150264 9375 ts_manager.cc:194] Registered new tserver with Master: b68afbeda46048cbb5ddbad465a98314 (127.28.243.67:34743)
I20260809 22:39:16.151706 9375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:33393
I20260809 22:39:16.159186 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:39:16.186348 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.165868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.28.243.126:39791)
0809 22:39:16.166221 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.166231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.167076 (+ 845us) server_negotiation.cc:407] Connection header received
0809 22:39:16.167332 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.167337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.167466 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.167580 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:16.170015 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.171195 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.173668 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.174483 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.174599 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.178477 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:16.178484 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:16.178489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:16.178529 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:16.182529 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.183570 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.183575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.183578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.183789 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.184292 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.184296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.184299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.184766 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:16.185051 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.185520 (+ 469us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.185798 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260809 22:39:16.191470 9375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:39:16.202097 9375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:39:16.235946 9747 heartbeater.cc:503] Master 127.28.243.126:39791 was elected leader, sending a full tablet report...
I20260809 22:39:16.237075 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.218707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:34559 (local address 127.0.0.1:33502)
0809 22:39:16.220856 (+ 2149us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:16.220908 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:16.221182 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:16.224933 (+ 3751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:16.224945 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:16.224994 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:39:16.226353 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.226370 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.229048 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.229071 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.232660 (+ 3589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.232679 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.235630 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.235640 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.235936 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.236022 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 22:39:16.236223 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 22:39:16.236457 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1688,"mutex_wait_us":35,"thread_start_us":4432,"threads_started":1}
I20260809 22:39:16.237124 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.220100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.28.243.66:38019)
0809 22:39:16.222321 (+ 2221us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.222329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.223758 (+ 1429us) server_negotiation.cc:407] Connection header received
0809 22:39:16.224103 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.224111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.224387 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.224723 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:39:16.228295 (+ 3572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.230233 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.233785 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.235177 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.235324 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.235598 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.236530 (+ 932us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.236744 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1994,"thread_start_us":2158,"threads_started":1}
I20260809 22:39:16.238148 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.219224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:38019 (local address 127.0.0.1:36760)
0809 22:39:16.223605 (+ 4381us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:16.223652 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:16.223839 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:16.225218 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:16.225223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:16.225254 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:39:16.228065 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.228079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.230427 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.230432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.233627 (+ 3195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.233639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.235926 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.235935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.236186 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.236217 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 22:39:16.237360 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0809 22:39:16.237508 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4204,"mutex_wait_us":23,"thread_start_us":1362,"threads_started":1}
I20260809 22:39:16.238541 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.218887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.28.243.65:34559)
0809 22:39:16.223219 (+ 4332us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.223226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.223251 (+ 25us) server_negotiation.cc:407] Connection header received
0809 22:39:16.223451 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.223458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.224538 (+ 1080us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.225424 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:39:16.226634 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.228909 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.232881 (+ 3972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.235243 (+ 2362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.237644 (+ 2401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.237871 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.237961 (+ 90us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.238156 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4118,"thread_start_us":3896,"threads_started":1}
I20260809 22:39:16.238907 9904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.223624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:34743 (local address 127.0.0.1:54326)
0809 22:39:16.228460 (+ 4836us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:16.228480 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:16.228669 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:16.229330 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:16.229339 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:16.229367 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:39:16.229937 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.229952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.232328 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.232334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.235328 (+ 2994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.235346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.238097 (+ 2751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.238104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.238298 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.238338 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 22:39:16.238448 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 22:39:16.238600 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4678,"thread_start_us":4388,"threads_started":1}
I20260809 22:39:16.239259 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.224342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.28.243.67:34743)
0809 22:39:16.227526 (+ 3184us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.227534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.228679 (+ 1145us) server_negotiation.cc:407] Connection header received
0809 22:39:16.228799 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.228837 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.229115 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.229326 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:39:16.230189 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.232123 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.235581 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.237490 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.237631 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.237871 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.238506 (+ 635us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.238830 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2964,"thread_start_us":2595,"threads_started":1}
I20260809 22:39:16.242455 9680 tablet_service.cc:1467] Processing CreateTablet for tablet 2181ff0282054ae4ae448427c98d7a62 (DEFAULT_TABLE table=test-table [id=7d9579edcde3441ba83ef50440f33462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:39:16.243749 9680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2181ff0282054ae4ae448427c98d7a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:39:16.243116 9531 tablet_service.cc:1467] Processing CreateTablet for tablet 2181ff0282054ae4ae448427c98d7a62 (DEFAULT_TABLE table=test-table [id=7d9579edcde3441ba83ef50440f33462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:39:16.243076 9819 tablet_service.cc:1467] Processing CreateTablet for tablet 2181ff0282054ae4ae448427c98d7a62 (DEFAULT_TABLE table=test-table [id=7d9579edcde3441ba83ef50440f33462]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:39:16.245085 9819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2181ff0282054ae4ae448427c98d7a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:39:16.245121 9531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2181ff0282054ae4ae448427c98d7a62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:39:16.256798 9906 tablet_bootstrap.cc:492] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Bootstrap starting.
I20260809 22:39:16.260174 9906 tablet_bootstrap.cc:654] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Neither blocks nor log segments found. Creating new log.
I20260809 22:39:16.261675 9906 log.cc:824] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Log is configured to *not* fsync() on all Append() calls
I20260809 22:39:16.262064 9907 tablet_bootstrap.cc:492] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Bootstrap starting.
I20260809 22:39:16.266390 9907 tablet_bootstrap.cc:654] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Neither blocks nor log segments found. Creating new log.
I20260809 22:39:16.266582 9906 tablet_bootstrap.cc:492] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: No bootstrap required, opened a new log
I20260809 22:39:16.266889 9906 ts_tablet_manager.cc:1397] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260809 22:39:16.267856 9907 log.cc:824] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Log is configured to *not* fsync() on all Append() calls
I20260809 22:39:16.269491 9909 tablet_bootstrap.cc:492] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Bootstrap starting.
I20260809 22:39:16.271987 9907 tablet_bootstrap.cc:492] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: No bootstrap required, opened a new log
I20260809 22:39:16.272307 9907 ts_tablet_manager.cc:1397] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260809 22:39:16.276028 9909 tablet_bootstrap.cc:654] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Neither blocks nor log segments found. Creating new log.
I20260809 22:39:16.277354 9909 log.cc:824] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Log is configured to *not* fsync() on all Append() calls
I20260809 22:39:16.278632 9906 raft_consensus.cc:348] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.279127 9906 raft_consensus.cc:374] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:39:16.279237 9906 raft_consensus.cc:729] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749cc4faf4cc4cfaa674aac3acab20fe, State: Initialized, Role: FOLLOWER
I20260809 22:39:16.279934 9906 consensus_queue.cc:260] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.281006 9909 tablet_bootstrap.cc:492] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: No bootstrap required, opened a new log
I20260809 22:39:16.281298 9909 ts_tablet_manager.cc:1397] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260809 22:39:16.283560 9906 ts_tablet_manager.cc:1428] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260809 22:39:16.283948 9907 raft_consensus.cc:348] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.284375 9907 raft_consensus.cc:374] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:39:16.284636 9907 raft_consensus.cc:729] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fde9703ec6a44cfb0181e07738a7c25, State: Initialized, Role: FOLLOWER
I20260809 22:39:16.285468 9907 consensus_queue.cc:260] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.287720 9907 ts_tablet_manager.cc:1428] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260809 22:39:16.295300 9909 raft_consensus.cc:348] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.295850 9909 raft_consensus.cc:374] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:39:16.296012 9909 raft_consensus.cc:729] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68afbeda46048cbb5ddbad465a98314, State: Initialized, Role: FOLLOWER
I20260809 22:39:16.296877 9909 consensus_queue.cc:260] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.298362 9887 heartbeater.cc:503] Master 127.28.243.126:39791 was elected leader, sending a full tablet report...
I20260809 22:39:16.299266 9909 ts_tablet_manager.cc:1428] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260809 22:39:16.355482 9888 tablet.cc:2369] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:39:16.399626 9914 raft_consensus.cc:482] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:39:16.400195 9914 raft_consensus.cc:504] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.402846 9914 leader_election.cc:288] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fde9703ec6a44cfb0181e07738a7c25 (127.28.243.65:34559), 749cc4faf4cc4cfaa674aac3acab20fe (127.28.243.66:38019)
I20260809 22:39:16.412374 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.403500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:38019 (local address 127.28.243.67:56895)
0809 22:39:16.403762 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:16.403786 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:16.403950 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:16.404709 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:16.404717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:16.404764 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:39:16.405467 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.405485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.407162 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.407166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.409853 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.409887 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.411645 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.411650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.411790 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.411827 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 22:39:16.411932 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 22:39:16.412091 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260809 22:39:16.412616 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.403886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:56895 (local address 127.28.243.66:38019)
0809 22:39:16.404200 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.404209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.404253 (+ 44us) server_negotiation.cc:407] Connection header received
0809 22:39:16.404363 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.404369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.404495 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.404668 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:39:16.405691 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.406987 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.410166 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.411456 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.411707 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.411796 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.412035 (+ 239us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.412187 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 22:39:16.414569 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.403799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:39483 (local address 127.28.243.65:34559)
0809 22:39:16.404177 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.404182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.405965 (+ 1783us) server_negotiation.cc:407] Connection header received
0809 22:39:16.406072 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.406077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.406196 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.406361 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:39:16.407140 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.408213 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.411417 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.413302 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.413564 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.413622 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.414185 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.414304 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260809 22:39:16.414314 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2181ff0282054ae4ae448427c98d7a62" candidate_uuid: "b68afbeda46048cbb5ddbad465a98314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" is_pre_election: true
I20260809 22:39:16.414968 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.404259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:34559 (local address 127.28.243.67:39483)
0809 22:39:16.405838 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0809 22:39:16.405853 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 22:39:16.405969 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:39:16.406346 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:39:16.406350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:39:16.406370 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:39:16.406902 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.406910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.408425 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.408432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.411158 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:39:16.411172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.413671 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.413678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:39:16.413967 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.414006 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 22:39:16.414117 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 22:39:16.414297 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1443,"thread_start_us":1117,"threads_started":1}
I20260809 22:39:16.415513 9701 raft_consensus.cc:2454] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68afbeda46048cbb5ddbad465a98314 in term 0.
I20260809 22:39:16.416545 9772 leader_election.cc:302] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749cc4faf4cc4cfaa674aac3acab20fe, b68afbeda46048cbb5ddbad465a98314; no voters:
I20260809 22:39:16.417173 9914 raft_consensus.cc:2781] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:39:16.417508 9914 raft_consensus.cc:482] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:39:16.416931 9552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2181ff0282054ae4ae448427c98d7a62" candidate_uuid: "b68afbeda46048cbb5ddbad465a98314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fde9703ec6a44cfb0181e07738a7c25" is_pre_election: true
I20260809 22:39:16.417733 9914 raft_consensus.cc:3037] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:39:16.418087 9552 raft_consensus.cc:2454] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b68afbeda46048cbb5ddbad465a98314 in term 0.
I20260809 22:39:16.421764 9914 raft_consensus.cc:504] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.422722 9914 leader_election.cc:288] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [CANDIDATE]: Term 1 election: Requested vote from peers 0fde9703ec6a44cfb0181e07738a7c25 (127.28.243.65:34559), 749cc4faf4cc4cfaa674aac3acab20fe (127.28.243.66:38019)
I20260809 22:39:16.423451 9552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2181ff0282054ae4ae448427c98d7a62" candidate_uuid: "b68afbeda46048cbb5ddbad465a98314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fde9703ec6a44cfb0181e07738a7c25"
I20260809 22:39:16.423530 9701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2181ff0282054ae4ae448427c98d7a62" candidate_uuid: "b68afbeda46048cbb5ddbad465a98314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749cc4faf4cc4cfaa674aac3acab20fe"
I20260809 22:39:16.423813 9552 raft_consensus.cc:3037] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:39:16.423874 9701 raft_consensus.cc:3037] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:39:16.427269 9701 raft_consensus.cc:2454] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68afbeda46048cbb5ddbad465a98314 in term 1.
I20260809 22:39:16.427516 9552 raft_consensus.cc:2454] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b68afbeda46048cbb5ddbad465a98314 in term 1.
I20260809 22:39:16.427839 9772 leader_election.cc:302] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749cc4faf4cc4cfaa674aac3acab20fe, b68afbeda46048cbb5ddbad465a98314; no voters:
I20260809 22:39:16.428299 9914 raft_consensus.cc:2781] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:39:16.429304 9914 raft_consensus.cc:686] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 1 LEADER]: Becoming Leader. State: Replica: b68afbeda46048cbb5ddbad465a98314, State: Running, Role: LEADER
I20260809 22:39:16.430204 9914 consensus_queue.cc:237] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } }
I20260809 22:39:16.436671 9374 catalog_manager.cc:5495] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 reported cstate change: term changed from 0 to 1, leader changed from <none> to b68afbeda46048cbb5ddbad465a98314 (127.28.243.67). New cstate: current_term: 1 leader_uuid: "b68afbeda46048cbb5ddbad465a98314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b68afbeda46048cbb5ddbad465a98314" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 34743 } health_report { overall_health: HEALTHY } } }
W20260809 22:39:16.448985 9748 tablet.cc:2369] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:39:16.475683 9600 tablet.cc:2369] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:39:16.481979 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.474904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.28.243.67:34743)
0809 22:39:16.475212 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.475221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.475247 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:39:16.475372 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.475379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.475513 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.475745 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:39:16.476381 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.477480 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.479290 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.479848 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.479950 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.480610 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:39:16.481176 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:39:16.481301 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.481508 (+ 207us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.481583 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:39:16.493032 9701 raft_consensus.cc:1261] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 1 FOLLOWER]: Refusing update from remote peer b68afbeda46048cbb5ddbad465a98314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:39:16.493350 9552 raft_consensus.cc:1261] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 1 FOLLOWER]: Refusing update from remote peer b68afbeda46048cbb5ddbad465a98314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:39:16.494154 9916 consensus_queue.cc:1035] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "749cc4faf4cc4cfaa674aac3acab20fe" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 22:39:16.496081 9914 consensus_queue.cc:1035] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fde9703ec6a44cfb0181e07738a7c25" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:39:16.512285 9919 mvcc.cc:205] Tried to move back new op lower bound from 7316746880982790144 to 7316746880743055360. Current Snapshot: MvccSnapshot[applied={T|T < 7316746880982790144}]
I20260809 22:39:16.514658 9921 mvcc.cc:205] Tried to move back new op lower bound from 7316746880982790144 to 7316746880743055360. Current Snapshot: MvccSnapshot[applied={T|T < 7316746880982790144}]
I20260809 22:39:16.539639 9374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:39:16.540133 9374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:39:16.545801 9374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f03be66b469461ca33d4c8c00944f23: Sending DeleteTablet for 3 replicas of tablet 2181ff0282054ae4ae448427c98d7a62
I20260809 22:39:16.547595 9680 tablet_service.cc:1514] Processing DeleteTablet for tablet 2181ff0282054ae4ae448427c98d7a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:39:16 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:36760
I20260809 22:39:16.547605 9531 tablet_service.cc:1514] Processing DeleteTablet for tablet 2181ff0282054ae4ae448427c98d7a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:39:16 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:33502
I20260809 22:39:16.547752 9819 tablet_service.cc:1514] Processing DeleteTablet for tablet 2181ff0282054ae4ae448427c98d7a62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:39:16 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:54326
I20260809 22:39:16.551007 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 9459
I20260809 22:39:16.562016 9932 tablet_replica.cc:330] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: stopping tablet replica
I20260809 22:39:16.562880 9932 raft_consensus.cc:2229] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:39:16.564054 9932 raft_consensus.cc:2258] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:39:16.567181 9930 tablet_replica.cc:330] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: stopping tablet replica
I20260809 22:39:16.567796 9930 raft_consensus.cc:2229] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:39:16.568637 9930 raft_consensus.cc:2258] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:39:16.569250 9931 tablet_replica.cc:330] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: stopping tablet replica
I20260809 22:39:16.569711 9931 raft_consensus.cc:2229] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:39:16.570351 9932 ts_tablet_manager.cc:1905] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:39:16.570452 9931 raft_consensus.cc:2258] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:39:16.573151 9931 ts_tablet_manager.cc:1905] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:39:16.583559 9931 ts_tablet_manager.cc:1918] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:39:16.583901 9931 log.cc:1196] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/2181ff0282054ae4ae448427c98d7a62
I20260809 22:39:16.584962 9931 ts_tablet_manager.cc:1939] T 2181ff0282054ae4ae448427c98d7a62 P 0fde9703ec6a44cfb0181e07738a7c25: Deleting consensus metadata
I20260809 22:39:16.585260 9930 ts_tablet_manager.cc:1905] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:39:16.587877 9362 catalog_manager.cc:4841] TS 0fde9703ec6a44cfb0181e07738a7c25 (127.28.243.65:34559): tablet 2181ff0282054ae4ae448427c98d7a62 (table test-table [id=7d9579edcde3441ba83ef50440f33462]) successfully deleted
I20260809 22:39:16.592952 9930 ts_tablet_manager.cc:1918] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:39:16.593156 9930 log.cc:1196] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/2181ff0282054ae4ae448427c98d7a62
I20260809 22:39:16.593129 9932 ts_tablet_manager.cc:1918] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:39:16.593431 9932 log.cc:1196] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/2181ff0282054ae4ae448427c98d7a62
I20260809 22:39:16.593730 9930 ts_tablet_manager.cc:1939] T 2181ff0282054ae4ae448427c98d7a62 P 749cc4faf4cc4cfaa674aac3acab20fe: Deleting consensus metadata
I20260809 22:39:16.594475 9932 ts_tablet_manager.cc:1939] T 2181ff0282054ae4ae448427c98d7a62 P b68afbeda46048cbb5ddbad465a98314: Deleting consensus metadata
Aug 09 22:39:16 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:39:16.596062 9361 catalog_manager.cc:4841] TS 749cc4faf4cc4cfaa674aac3acab20fe (127.28.243.66:38019): tablet 2181ff0282054ae4ae448427c98d7a62 (table test-table [id=7d9579edcde3441ba83ef50440f33462]) successfully deleted
I20260809 22:39:16.598762 9362 catalog_manager.cc:4841] TS b68afbeda46048cbb5ddbad465a98314 (127.28.243.67:34743): tablet 2181ff0282054ae4ae448427c98d7a62 (table test-table [id=7d9579edcde3441ba83ef50440f33462]) successfully deleted
I20260809 22:39:16.609841 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.554111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.28.243.65:34559)
0809 22:39:16.554365 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.554377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.565738 (+ 11361us) server_negotiation.cc:407] Connection header received
0809 22:39:16.566028 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.566034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.566224 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.566432 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:16.577437 (+ 11005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.579048 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.582218 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.583333 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.583517 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.598112 (+ 14595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:16.598133 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:16.598153 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:16.598233 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:16.604472 (+ 6239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.605960 (+ 1488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.605971 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.605976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.606212 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.606786 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.606792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.606796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.607474 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:16.607842 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.608956 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.609242 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 22:39:16.610755 9594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:39:16.703738 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 9459
I20260809 22:39:16.724196 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 9605
Aug 09 22:39:16 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:39:16.741729 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.725204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.28.243.66:38019)
0809 22:39:16.725442 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.725448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.725491 (+ 43us) server_negotiation.cc:407] Connection header received
0809 22:39:16.725553 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.725557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.725632 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.725743 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:16.727100 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.727865 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.729249 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.729700 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.729841 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.734803 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:16.734820 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:16.734836 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:16.734877 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:16.738429 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.739290 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.739297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.739300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.739439 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.739846 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.739852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.739855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.740249 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:16.740627 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.741061 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.741324 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:39:16.742825 9741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:39:16.839078 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 9605
I20260809 22:39:16.860174 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 9753
Aug 09 22:39:16 dist-test-slave-rgb1 krb5kdc[8869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:39:16.879805 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:39:16.861049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.28.243.67:34743)
0809 22:39:16.861262 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 22:39:16.861268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:39:16.861294 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:39:16.861402 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:39:16.861409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:39:16.861498 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:39:16.861638 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:39:16.863108 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.864309 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.865978 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:39:16.866683 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:39:16.866844 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:39:16.872826 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:39:16.872840 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:39:16.872858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:39:16.872916 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:39:16.876404 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.877450 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.877459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.877461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.877669 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:39:16.878196 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:39:16.878200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:39:16.878202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:39:16.878590 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:39:16.878854 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 22:39:16.879245 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 22:39:16.879450 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 22:39:16.880721 9882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:39:17.010680 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 9753
I20260809 22:39:17.032894 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 9342
I20260809 22:39:17.034340 9406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:39:17.152275 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 9342
I20260809 22:39:17.180547 29645 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 22:39:17.527123 29645 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 22:39:17.527426 29645 mini_ranger.cc:66] Stopping Ranger...
I20260809 22:39:17.874186 29645 mini_ranger.cc:68] Stopped Ranger
2026-08-09 22:39:17.874 UTC [8902] LOG: received smart shutdown request
2026-08-09 22:39:17.879 UTC [8902] LOG: background worker "logical replication launcher" (PID 8912) exited with exit code 1
2026-08-09 22:39:17.879 UTC [8907] LOG: shutting down
2026-08-09 22:39:18.122 UTC [8902] LOG: database system is shut down
2026-08-09T22:39:18Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (56366 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:39:18 dist-test-slave-rgb1 krb5kdc[9947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:39:18 dist-test-slave-rgb1 krb5kdc[9947](info): set up 2 sockets
Aug 09 22:39:18 dist-test-slave-rgb1 krb5kdc[9947](info): commencing operation
krb5kdc: starting...
W20260809 22:39:19.518218 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:39:19 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:39:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:39:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/rangeradmin_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/rangeradmin_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/rangerlookup_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/rangerlookup_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
I20260809 22:39:19.952713 29645 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmuNQXR/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/postgres -l logfile start
2026-08-09 22:39:23.126 UTC [9978] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 22:39:23.126 UTC [9978] LOG: listening on IPv4 address "127.28.243.84", port 40707
2026-08-09 22:39:23.131 UTC [9978] LOG: listening on Unix socket "/tmp/.s.PGSQL.40707"
2026-08-09 22:39:23.157 UTC [9981] LOG: database system was shut down at 2026-08-09 22:39:21 UTC
2026-08-09 22:39:23.161 UTC [9978] LOG: database system is ready to accept connections
I20260809 22:39:24.221354 29645 mini_postgres.cc:96] Postgres bound to 40707
2026-08-09 22:39:24.245 UTC [9989] FATAL: database "slave" does not exist
127.28.243.84:40707 - accepting connections
I20260809 22:39:24.246641 29645 mini_ranger.cc:161] Starting Ranger...
I20260809 22:39:24.280642 29645 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 22:39:24.984157 29645 mini_ranger.cc:87] Created ranger Postgres database
I20260809 22:39:24.984290 29645 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-admin
2026-08-09 22:39:25,311 [I] DB FLAVOR :POSTGRES
2026-08-09 22:39:25,312 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 22:39:25,312 [I] Checking connection..
2026-08-09 22:39:25,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 22:39:25,605 [I] Checking connection passed.
2026-08-09 22:39:25,606 [I] --------- Verifying version history table ---------
2026-08-09 22:39:25,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 22:39:25,881 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 22:39:25,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 22:39:26,179 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 22:39:26,179 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 22:39:26,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 22:39:26.434 UTC [10054] WARNING: there is no transaction in progress
2026-08-09 22:39:26,442 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 22:39:26,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 22:39:26,711 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 22:39:26,711 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 22:39:26,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 22:39:26,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 22:39:27,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rgb1.c.gcp-upstream.internal','N') ;"
2026-08-09 22:39:27,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 22:39:27,767 [I] Table x_portal_user does not exist in database ranger
2026-08-09 22:39:27,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 22:39:28,019 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 22:39:28,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 22:39:28,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 22:39:28,561 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 22:39:28,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 22:39:29.690 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.710 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.729 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.748 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.767 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.789 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.848 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.857 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.875 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.886 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.898 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.913 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.925 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.934 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.943 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:29.951 UTC [10166] WARNING: there is no transaction in progress
2026-08-09 22:39:30,837 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 22:39:30,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 22:39:31,094 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 22:39:31,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 22:39:31,424 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 22:39:31,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 22:39:31,695 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 22:39:31,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 22:39:31,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 22:39:32,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rgb1.c.gcp-upstream.internal';"
2026-08-09 22:39:32,496 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 22:39:32,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 22:39:32,741 [I] DB_PATCHES have already been applied
I20260809 22:39:32.747296 29645 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmuNQXR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 22:39:32.747401 29645 mini_ranger.cc:193] Using host: 127.28.243.84
I20260809 22:39:32.769351 29645 mini_ranger.cc:227] Ranger admin URL: http://127.28.243.84:60965
Aug 09, 2026 10:39:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 10:39:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 10:39:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 10:39:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 10:39:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 10:39:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 10:39:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 22:39:33.993014 29645 mini_ranger.cc:160] Time spent starting Ranger: real 9.746s user 0.000s sys 0.064s
Aug 09, 2026 10:39:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 10:39:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 10:39:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 10:39:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 10:39:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 22:39:56.772286 29645 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.243.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.243.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/rangerkms_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.243.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/rangerkms_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.243.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.243.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.243.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/HTTP_127.28.243.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 22:39:56 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315196, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 22:39:56.995810 29645 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 22:39:57.025182 29645 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 22:39:58.147490 29645 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 22:39:58.147645 29645 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms
I20260809 22:39:58.147717 29645 mini_ranger_kms.cc:227] Using postgres at 127.28.243.84:40707
2026-08-09 22:39:58,545 [I] DB FLAVOR :POSTGRES
2026-08-09 22:39:58,546 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 22:39:58,546 [I] Checking connection
2026-08-09 22:39:58,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 22:39:58,841 [I] connection success
2026-08-09 22:39:58,841 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 22:39:58,841 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 22:39:58,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 22:39:59,102 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 22:39:59,103 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 22:39:59,103 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 22:39:59,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.243.84:40707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 22:39:59,422 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 22:39:59.922107 29645 mini_ranger_kms.cc:310] Created kms service
I20260809 22:40:00.125943 29645 mini_ranger_kms.cc:326] Created kudu user
I20260809 22:40:00.188265 29645 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 22:40:00 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315196, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:00.671094 29645 mini_ranger_kms.cc:384] Added ranger policy
I20260809 22:40:00.671456 29645 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmuNQXR/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmuNQXR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 22:40:00.671528 29645 mini_ranger_kms.cc:240] Using host: 127.28.243.84
I20260809 22:40:00.700148 29645 mini_ranger_kms.cc:276] Ranger KMS PID: 10354
I20260809 22:40:00.700330 29645 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.243.84:58207
Aug 09, 2026 10:40:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 10:40:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 10:40:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 10:40:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 10:40:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 10:40:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260809 22:40:03.234404 29645 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.239s user 0.053s sys 0.385s
I20260809 22:40:03.234683 29645 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 22:40:03.234867 29645 mini_ranger_kms.cc:401] 127.28.243.84:58207/kms/v1/keys
Aug 09 22:40:03 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786315203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 22:40:03 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.84@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
Aug 09 22:40:04 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.243.84@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
Aug 09 22:40:06 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:40:07.004323 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35099
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:35099
--ranger_config_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:07.368791 10408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:07.369167 10408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:07.369316 10408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:07.376951 10408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:07.377079 10408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:07.377127 10408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:07.377255 10408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:07.377342 10408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:07.388110 10408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.243.84:58207/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:35099
--ranger_config_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:07.389605 10408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:07.390985 10408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:07.407754 10414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:07.408255 10416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:07.408861 10413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:07.410189 10408 server_base.cc:1034] running on GCE node
I20260809 22:40:07.411206 10408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:07.413000 10408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:07.414268 10408 hybrid_clock.cc:648] HybridClock initialized: now 1786315207414195 us; error 73 us; skew 500 ppm
I20260809 22:40:07.414691 10408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:07 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:07.418524 10408 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:07.420229 10408 webserver.cc:460] Webserver started at http://127.28.243.126:36333/ using document root <none> and password file <none>
I20260809 22:40:07.421037 10408 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:07.421203 10408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:07.421564 10408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:40:07 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315207, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:07.489683 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "4771d126e81f45c5b234466b78fc897f"
format_stamp: "Formatted at 2026-08-09 22:40:07 on dist-test-slave-rgb1"
server_key: "1de4c2d6431fed6d0de9350114463a38"
server_key_iv: "47b258f9fd6a0cc1027132dad77400c5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:07.490839 10408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "4771d126e81f45c5b234466b78fc897f"
format_stamp: "Formatted at 2026-08-09 22:40:07 on dist-test-slave-rgb1"
server_key: "1de4c2d6431fed6d0de9350114463a38"
server_key_iv: "47b258f9fd6a0cc1027132dad77400c5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:07.497005 10408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 22:40:07.514925 10423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:07.516727 10408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:40:07.516948 10408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "4771d126e81f45c5b234466b78fc897f"
format_stamp: "Formatted at 2026-08-09 22:40:07 on dist-test-slave-rgb1"
server_key: "1de4c2d6431fed6d0de9350114463a38"
server_key_iv: "47b258f9fd6a0cc1027132dad77400c5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:07.517191 10408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:07.531622 10408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:07.536751 10408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:07.537101 10408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:07.561487 10408 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:35099
I20260809 22:40:07.561518 10479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:35099 every 8 connection(s)
I20260809 22:40:07.563066 10408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:07.567996 10480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:07.571319 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 10408
I20260809 22:40:07.571609 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:40:07.572494 29645 external_mini_cluster.cc:1428] Setting key 37cee8fc6935c74727c31f2b3e6c1012
I20260809 22:40:07.584184 10480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Bootstrap starting.
I20260809 22:40:07.589036 10480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Neither blocks nor log segments found. Creating new log.
Aug 09 22:40:07 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:07.590353 10480 log.cc:824] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:07.593725 10480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: No bootstrap required, opened a new log
I20260809 22:40:07.599570 10483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:07.575161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.28.243.126:35099)
0809 22:40:07.576264 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:07.576283 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:07.576333 (+ 50us) server_negotiation.cc:407] Connection header received
0809 22:40:07.578386 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:07.578414 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:07.578843 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:07.579213 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:07.581088 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:07.582675 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:07.584724 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:07.585540 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:07.585705 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:07.591603 (+ 5898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:07.591614 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:07.591631 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:07.591674 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:07.594959 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:07.595779 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:07.595787 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:07.595793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:07.595971 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:07.596426 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:07.596430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:07.596432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:07.597084 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:07.597438 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:07.597938 (+ 500us) server_negotiation.cc:299] Negotiation successful
0809 22:40:07.598461 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":572,"threads_started":1}
I20260809 22:40:07.602241 10480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:07.602648 10480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:07.602771 10480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4771d126e81f45c5b234466b78fc897f, State: Initialized, Role: FOLLOWER
I20260809 22:40:07.603336 10480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:07.603597 10480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:40:07.603749 10480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:40:07.603931 10480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:07.606353 10480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:07.606871 10480 leader_election.cc:302] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4771d126e81f45c5b234466b78fc897f; no voters:
I20260809 22:40:07.607530 10480 leader_election.cc:288] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:40:07.607672 10485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:07.608692 10485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 LEADER]: Becoming Leader. State: Replica: 4771d126e81f45c5b234466b78fc897f, State: Running, Role: LEADER
I20260809 22:40:07.609360 10485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:07.610126 10480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:07.612623 10487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4771d126e81f45c5b234466b78fc897f. Latest consensus state: current_term: 1 leader_uuid: "4771d126e81f45c5b234466b78fc897f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } } }
I20260809 22:40:07.613096 10487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:07.613142 10486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4771d126e81f45c5b234466b78fc897f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } } }
I20260809 22:40:07.613479 10486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:07.615471 10491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:07.621502 10491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:07.622403 10480 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 22:40:07.645880 10491 catalog_manager.cc:1347] Generated new cluster ID: 049a69756bb64319ad7b26045bfc5af0
I20260809 22:40:07.646077 10491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:40:07.686638 10491 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:40:07.688908 10491 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:40:07.702160 10491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Generated new TSK 0
I20260809 22:40:07.703325 10491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:40:08.509830 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:35099
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:40:08 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786315208, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 22:40:08.959650 10526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:08.960024 10526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:08.960217 10526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:08.968712 10526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:08.968879 10526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:08.969086 10526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:40:08.980823 10526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.243.84:58207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:08.982360 10526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:08.983758 10526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:09.005025 10538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:09.006032 10537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:09.008375 10540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:09.008956 10526 server_base.cc:1034] running on GCE node
I20260809 22:40:09.010133 10526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:09.011781 10526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:09.015285 10526 hybrid_clock.cc:648] HybridClock initialized: now 1786315209015164 us; error 127 us; skew 500 ppm
I20260809 22:40:09.015862 10526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:09 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315209, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:09.021301 10526 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:40:09.026896 10526 webserver.cc:460] Webserver started at http://127.28.243.65:44055/ using document root <none> and password file <none>
I20260809 22:40:09.027781 10526 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:09.027949 10526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:09.028321 10526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:40:09 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315209, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:09.049826 10526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "a96c4f201d55423c98cb307a5bb38f83"
format_stamp: "Formatted at 2026-08-09 22:40:09 on dist-test-slave-rgb1"
server_key: "4b87ade48af256f355933194a1087913"
server_key_iv: "b02bd4113afe34aa0c779478f86634e6"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:09.051026 10526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "a96c4f201d55423c98cb307a5bb38f83"
format_stamp: "Formatted at 2026-08-09 22:40:09 on dist-test-slave-rgb1"
server_key: "4b87ade48af256f355933194a1087913"
server_key_iv: "b02bd4113afe34aa0c779478f86634e6"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:09.064997 10526 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.009s
I20260809 22:40:09.099880 10547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:09.102253 10526 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.007s sys 0.008s
I20260809 22:40:09.102562 10526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "a96c4f201d55423c98cb307a5bb38f83"
format_stamp: "Formatted at 2026-08-09 22:40:09 on dist-test-slave-rgb1"
server_key: "4b87ade48af256f355933194a1087913"
server_key_iv: "b02bd4113afe34aa0c779478f86634e6"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:09.102932 10526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:09.154891 10526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:09.160112 10526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:09.160518 10526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:09.161761 10526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:09.163779 10526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:09.163945 10526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 22:40:09.164125 10526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:09.164206 10526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:09.253054 10526 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:38441
I20260809 22:40:09.253105 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:38441 every 8 connection(s)
I20260809 22:40:09.255234 10526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:40:09.257793 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 10526
I20260809 22:40:09.258037 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:40:09.258965 29645 external_mini_cluster.cc:1428] Setting key 61ad87cea0d87cd97fb91bbe8b225339
Aug 09 22:40:09 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315209, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:09.299443 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:09.270113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:39079 (local address 127.28.243.126:35099)
0809 22:40:09.271113 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:09.271124 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:09.271917 (+ 793us) server_negotiation.cc:407] Connection header received
0809 22:40:09.275711 (+ 3794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:09.275720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:09.276034 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:09.276192 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:09.279129 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:09.280511 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:09.283471 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:09.284195 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:09.284314 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:09.290390 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:09.290400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:09.290406 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:09.290459 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:09.294751 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:09.296015 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:09.296022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:09.296039 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:09.296223 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:09.296903 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:09.296912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:09.296920 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:09.297374 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:09.297530 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:09.298701 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0809 22:40:09.298979 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":439,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
I20260809 22:40:09.302580 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:09.270401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:35099 (local address 127.28.243.65:39079)
0809 22:40:09.271740 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:09.271790 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:09.274437 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:09.277044 (+ 2607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:09.277060 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:09.277864 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:09.278829 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:09.278859 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:09.280824 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:09.280830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:09.283216 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:09.283247 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:09.284794 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:09.284803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:09.285167 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:09.286139 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:09.286174 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:09.289944 (+ 3770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:09.295051 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:09.295064 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:09.295088 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:09.295838 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:09.296404 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:09.296409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:09.296412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:09.296745 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:09.297651 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:09.297660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:09.298195 (+ 535us) client_negotiation.cc:769] Sending connection context
0809 22:40:09.300111 (+ 1916us) client_negotiation.cc:240] Negotiation successful
0809 22:40:09.300815 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"thread_start_us":806,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
I20260809 22:40:09.305047 10666 heartbeater.cc:347] Connected to a master server at 127.28.243.126:35099
I20260809 22:40:09.305418 10666 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:09.306437 10666 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
I20260809 22:40:09.309476 10441 ts_manager.cc:194] Registered new tserver with Master: a96c4f201d55423c98cb307a5bb38f83 (127.28.243.65:38441)
I20260809 22:40:09.311801 10441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:39079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:40:09.354864 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:35099
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:40:09 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315208, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
Aug 09 22:40:09 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315208, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
W20260809 22:40:09.914458 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:09.915030 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:09.915249 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:09.927589 10673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:09.927855 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:09.928123 10673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:40:09.944381 10673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.243.84:58207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:09.946604 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:09.948594 10673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:09.989464 10682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:09.990338 10681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:09.992750 10684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:09.993448 10673 server_base.cc:1034] running on GCE node
I20260809 22:40:09.994371 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:09.997387 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:09.998742 10673 hybrid_clock.cc:648] HybridClock initialized: now 1786315209998657 us; error 86 us; skew 500 ppm
I20260809 22:40:09.999285 10673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:09 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315209, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:10.004594 10673 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:40:10.012194 10673 webserver.cc:460] Webserver started at http://127.28.243.66:43831/ using document root <none> and password file <none>
I20260809 22:40:10.013952 10673 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:10.014307 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:10.015211 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:40:10 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315209, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:10.040657 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "d076022427094727b54165074ea5ddc7"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "ff32a450655673df17087cc7919d00e5"
server_key_iv: "ed80f873ba928f4a3995f02f904a3f3b"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:10.041970 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "d076022427094727b54165074ea5ddc7"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "ff32a450655673df17087cc7919d00e5"
server_key_iv: "ed80f873ba928f4a3995f02f904a3f3b"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:10.051666 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260809 22:40:10.079459 10693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:10.083230 10673 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.000s
I20260809 22:40:10.083639 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "d076022427094727b54165074ea5ddc7"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "ff32a450655673df17087cc7919d00e5"
server_key_iv: "ed80f873ba928f4a3995f02f904a3f3b"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:10.084048 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:10.104384 10673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:10.109694 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:10.110046 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:10.111359 10673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:10.113066 10673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:10.113145 10673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:10.113298 10673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:10.113349 10673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:10.162784 10673 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:35885
I20260809 22:40:10.162787 10817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:35885 every 8 connection(s)
I20260809 22:40:10.164160 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:40:10.170292 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 10673
I20260809 22:40:10.170496 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:40:10.171087 29645 external_mini_cluster.cc:1428] Setting key d5188e7a4f7c59f53d2256edbbb72acf
Aug 09 22:40:10 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315209, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:10.196858 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:10.168175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:42131 (local address 127.28.243.126:35099)
0809 22:40:10.168727 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:10.168732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:10.171838 (+ 3106us) server_negotiation.cc:407] Connection header received
0809 22:40:10.172841 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:10.172847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:10.172952 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:10.173086 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:10.176707 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.177923 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.180680 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.181410 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.181613 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:10.186509 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:10.186515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:10.186519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:10.186557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:10.192435 (+ 5878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:10.193713 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:10.193720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:10.193723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:10.193895 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:10.194501 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:10.194506 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:10.194509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:10.194935 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:10.195106 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:10.196114 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0809 22:40:10.196392 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":287,"threads_started":1}
I20260809 22:40:10.198244 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:10.169061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:35099 (local address 127.28.243.66:42131)
0809 22:40:10.170038 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:10.170096 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:10.172490 (+ 2394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:10.173329 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:10.173338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:10.173940 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:10.174813 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:10.174842 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.178152 (+ 3310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.178158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:10.180439 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:10.180460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.181704 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.181733 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:10.182071 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:10.183016 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:10.183050 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:10.186145 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:10.192707 (+ 6562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:10.192721 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:10.192743 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:10.193518 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:10.194040 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:10.194048 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:10.194055 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:10.194346 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:10.195103 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:10.195115 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:10.195631 (+ 516us) client_negotiation.cc:769] Sending connection context
0809 22:40:10.195965 (+ 334us) client_negotiation.cc:240] Negotiation successful
0809 22:40:10.196665 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":396,"threads_started":1}
I20260809 22:40:10.200062 10818 heartbeater.cc:347] Connected to a master server at 127.28.243.126:35099
I20260809 22:40:10.200416 10818 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:10.201355 10818 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
I20260809 22:40:10.203243 10441 ts_manager.cc:194] Registered new tserver with Master: d076022427094727b54165074ea5ddc7 (127.28.243.66:35885)
I20260809 22:40:10.204500 10441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:42131
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:40:10.245412 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:35099
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:10.316360 10666 heartbeater.cc:503] Master 127.28.243.126:35099 was elected leader, sending a full tablet report...
W20260809 22:40:10.630304 10826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:10.630668 10826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:10.630847 10826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:10.638366 10826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:10.638540 10826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:10.638729 10826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:40:10.650700 10826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.243.84:58207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:10.652128 10826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:10.653596 10826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:10.669385 10832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:10.669286 10833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:10.669652 10835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:10.670902 10826 server_base.cc:1034] running on GCE node
I20260809 22:40:10.671823 10826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:10.673153 10826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:10.674389 10826 hybrid_clock.cc:648] HybridClock initialized: now 1786315210674335 us; error 44 us; skew 500 ppm
I20260809 22:40:10.674774 10826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:10 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315210, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:10.683076 10826 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:40:10.684892 10826 webserver.cc:460] Webserver started at http://127.28.243.67:36615/ using document root <none> and password file <none>
I20260809 22:40:10.685698 10826 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:10.685820 10826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:10.686190 10826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 22:40:10 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315210, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:10.706300 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "11626be3728244f6a3c3143d41dddaff"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "2e4895f503f2234c6f5b10e206d9da0c"
server_key_iv: "d7de25b429f321b2f51460afea34abf5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:10.707646 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "11626be3728244f6a3c3143d41dddaff"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "2e4895f503f2234c6f5b10e206d9da0c"
server_key_iv: "d7de25b429f321b2f51460afea34abf5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:10.717238 10826 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20260809 22:40:10.733671 10844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:10.735592 10826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:40:10.735800 10826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "11626be3728244f6a3c3143d41dddaff"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "2e4895f503f2234c6f5b10e206d9da0c"
server_key_iv: "d7de25b429f321b2f51460afea34abf5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:10.736066 10826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:10.764537 10826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:10.769629 10826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:10.770145 10826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:10.771732 10826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:10.774227 10826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:10.774358 10826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:10.774485 10826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:10.774554 10826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:10.815090 10826 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:46247
I20260809 22:40:10.815097 10960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:46247 every 8 connection(s)
I20260809 22:40:10.816671 10826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:40:10.823549 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 10826
I20260809 22:40:10.823789 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:40:10.824542 29645 external_mini_cluster.cc:1428] Setting key 0462bfdf29d8096645713ac82cf3f026
Aug 09 22:40:10 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315210, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:10.842216 10963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:10.820080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:54227 (local address 127.28.243.126:35099)
0809 22:40:10.820717 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:10.820726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:10.822025 (+ 1299us) server_negotiation.cc:407] Connection header received
0809 22:40:10.824119 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:10.824127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:10.824232 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:10.824449 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:10.826361 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.827542 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.829722 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.830336 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.830452 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:10.834245 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:10.834373 (+ 128us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:10.834382 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:10.834435 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:10.837648 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:10.838593 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:10.838600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:10.838603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:10.838840 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:10.839485 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:10.839490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:10.839492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:10.839810 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:10.840027 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:10.841439 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0809 22:40:10.841742 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":278,"threads_started":1}
I20260809 22:40:10.843605 10964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:10.820628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:35099 (local address 127.28.243.67:54227)
0809 22:40:10.821738 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:10.821789 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:10.823699 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:10.824747 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:10.824756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:10.825338 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:10.826046 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:10.826064 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.827727 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.827732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:10.829543 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:10.829557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.830575 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.830581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:10.830808 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:10.831567 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:10.831591 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:10.833910 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:10.837821 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:10.837829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:10.837844 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:10.838400 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:10.839051 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:10.839055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:10.839058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:10.839288 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:10.840004 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:10.840012 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:10.840534 (+ 522us) client_negotiation.cc:769] Sending connection context
0809 22:40:10.840933 (+ 399us) client_negotiation.cc:240] Negotiation successful
0809 22:40:10.841757 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":528,"threads_started":1}
I20260809 22:40:10.845353 10961 heartbeater.cc:347] Connected to a master server at 127.28.243.126:35099
I20260809 22:40:10.845697 10961 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:10.846444 10961 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
I20260809 22:40:10.848203 10441 ts_manager.cc:194] Registered new tserver with Master: 11626be3728244f6a3c3143d41dddaff (127.28.243.67:46247)
I20260809 22:40:10.849251 10441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:54227
I20260809 22:40:10.856433 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:40:10.858424 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 10526
Aug 09 22:40:10 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:40:10.880498 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:10.859678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.28.243.65:38441)
0809 22:40:10.860645 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:10.860653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:10.860682 (+ 29us) server_negotiation.cc:407] Connection header received
0809 22:40:10.860758 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:10.860765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:10.860993 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:10.861143 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:10.862870 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.864334 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.866368 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.867079 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:10.867214 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:10.872731 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:10.872749 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:10.872766 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:10.872811 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:10.876395 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:10.877477 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:10.877486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:10.877488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:10.877698 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:10.878231 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:10.878235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:10.878237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:10.878763 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:10.879172 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:10.879844 (+ 672us) server_negotiation.cc:299] Negotiation successful
0809 22:40:10.880103 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":548,"threads_started":1}
I20260809 22:40:10.882184 10661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:10.972190 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 10526
I20260809 22:40:10.993651 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 10673
Aug 09 22:40:11 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:40:11.015174 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:10.994959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.28.243.66:35885)
0809 22:40:10.995557 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:10.995565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:10.995601 (+ 36us) server_negotiation.cc:407] Connection header received
0809 22:40:10.995661 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:10.995667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:10.995900 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:10.996081 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:10.997760 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:10.998906 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:11.000421 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:11.001256 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:11.001373 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:11.007563 (+ 6190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:11.007581 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:11.007615 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:11.007683 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:11.011565 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:11.012711 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:11.012724 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:11.012727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:11.012928 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:11.013426 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:11.013430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:11.013432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:11.013812 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:11.014122 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:11.014444 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 22:40:11.014691 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":308,"threads_started":1}
I20260809 22:40:11.017133 10812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:11.109337 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 10673
I20260809 22:40:11.135342 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 10826
Aug 09 22:40:11 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:40:11.158839 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:11.136870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.28.243.67:46247)
0809 22:40:11.137713 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:11.137721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:11.137755 (+ 34us) server_negotiation.cc:407] Connection header received
0809 22:40:11.137821 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:11.137828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:11.138051 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:11.138204 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:11.140136 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:11.142008 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:11.143893 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:11.144768 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:11.144906 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:11.150468 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:11.150491 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:11.150512 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:11.150576 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:11.154407 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:11.155769 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:11.155780 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:11.155783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:11.155996 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:11.156696 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:11.156710 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:11.156716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:11.157338 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:11.157642 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:11.158211 (+ 569us) server_negotiation.cc:299] Negotiation successful
0809 22:40:11.158485 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":575,"threads_started":1}
I20260809 22:40:11.160445 10956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:11.255127 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 10826
I20260809 22:40:11.275758 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 10408
I20260809 22:40:11.278548 10474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:11.396657 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 10408
I20260809 22:40:11.431640 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35099
--webserver_interface=127.28.243.126
--webserver_port=36333
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:35099
--ranger_config_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:11.809362 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:11.809743 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:11.809929 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:11.818357 10976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:11.818522 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:11.818642 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:11.818740 10976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:11.818867 10976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:11.829638 10976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:35099
--ranger_config_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=36333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:11.831092 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:11.832465 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:11.845471 10982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:11.845440 10983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:11.845412 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:11.846377 10976 server_base.cc:1034] running on GCE node
I20260809 22:40:11.847240 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:11.848827 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:11.850142 10976 hybrid_clock.cc:648] HybridClock initialized: now 1786315211850089 us; error 58 us; skew 500 ppm
I20260809 22:40:11.850567 10976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:11 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:11.854457 10976 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:11.856364 10976 webserver.cc:460] Webserver started at http://127.28.243.126:36333/ using document root <none> and password file <none>
I20260809 22:40:11.857205 10976 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:11.857393 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 22:40:11 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315211, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:11.912844 10976 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 22:40:11.916221 10995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:11.917768 10976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:40:11.917979 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "4771d126e81f45c5b234466b78fc897f"
format_stamp: "Formatted at 2026-08-09 22:40:07 on dist-test-slave-rgb1"
server_key: "1de4c2d6431fed6d0de9350114463a38"
server_key_iv: "47b258f9fd6a0cc1027132dad77400c5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:11.918987 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:11.936415 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:11.941130 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:11.941511 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:11.962666 10976 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:35099
I20260809 22:40:11.962723 11050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:35099 every 8 connection(s)
I20260809 22:40:11.964519 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:11.966290 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 10976
I20260809 22:40:11.967458 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:38441
--local_ip_for_outbound_sockets=127.28.243.65
--tserver_master_addrs=127.28.243.126:35099
--webserver_port=44055
--webserver_interface=127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:11.975831 11051 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:40:11.981000 11051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Bootstrap starting.
I20260809 22:40:12.002943 11051 log.cc:824] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:12.016609 11051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:40:12.017376 11051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Bootstrap complete.
I20260809 22:40:12.028535 11051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:12.029590 11051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4771d126e81f45c5b234466b78fc897f, State: Initialized, Role: FOLLOWER
I20260809 22:40:12.030361 11051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:12.030620 11051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:40:12.030769 11051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:40:12.031467 11051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:40:12.036863 11051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:12.037460 11051 leader_election.cc:302] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4771d126e81f45c5b234466b78fc897f; no voters:
I20260809 22:40:12.038271 11051 leader_election.cc:288] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 22:40:12.038515 11055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:40:12.040097 11055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [term 2 LEADER]: Becoming Leader. State: Replica: 4771d126e81f45c5b234466b78fc897f, State: Running, Role: LEADER
I20260809 22:40:12.040838 11055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } }
I20260809 22:40:12.041213 11051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:12.043776 11057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4771d126e81f45c5b234466b78fc897f. Latest consensus state: current_term: 2 leader_uuid: "4771d126e81f45c5b234466b78fc897f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } } }
I20260809 22:40:12.044297 11057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:12.044958 11056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4771d126e81f45c5b234466b78fc897f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4771d126e81f45c5b234466b78fc897f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35099 } } }
I20260809 22:40:12.045290 11056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:12.060699 11061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:12.067193 11061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:12.068840 11061 catalog_manager.cc:1259] Loaded cluster ID: 049a69756bb64319ad7b26045bfc5af0
I20260809 22:40:12.069029 11061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:40:12.073890 11061 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:40:12.076527 11061 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Loaded TSK: 0
I20260809 22:40:12.078467 11061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:40:12.082080 11051 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260809 22:40:12.456321 11053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:12.457473 11053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:12.457729 11053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:12.474081 11053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:12.474797 11053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:12.475428 11053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:40:12.495735 11053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:38441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=44055
--tserver_master_addrs=127.28.243.126:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:12.497833 11053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:12.501755 11053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:12.547276 11089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:12.547381 11090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:12.555562 11092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:12.556778 11053 server_base.cc:1034] running on GCE node
I20260809 22:40:12.557722 11053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:12.559275 11053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:12.560913 11053 hybrid_clock.cc:648] HybridClock initialized: now 1786315212560845 us; error 64 us; skew 500 ppm
I20260809 22:40:12.561424 11053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:12 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:12.566224 11053 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:40:12.579528 11053 webserver.cc:460] Webserver started at http://127.28.243.65:44055/ using document root <none> and password file <none>
I20260809 22:40:12.580801 11053 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:12.581034 11053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 22:40:12 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:12.618676 11053 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.005s
I20260809 22:40:12.624112 11101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:12.627285 11053 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260809 22:40:12.627521 11053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "a96c4f201d55423c98cb307a5bb38f83"
format_stamp: "Formatted at 2026-08-09 22:40:09 on dist-test-slave-rgb1"
server_key: "4b87ade48af256f355933194a1087913"
server_key_iv: "b02bd4113afe34aa0c779478f86634e6"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:12.629122 11053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:12.752821 11053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:12.758934 11053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:12.759327 11053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:12.765506 11053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:12.772712 11053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:12.772954 11053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:12.773149 11053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:12.773280 11053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:12.884251 11053 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:38441
I20260809 22:40:12.884325 11223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:38441 every 8 connection(s)
I20260809 22:40:12.885974 11053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:40:12.893728 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 11053
I20260809 22:40:12.894622 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:35885
--local_ip_for_outbound_sockets=127.28.243.66
--tserver_master_addrs=127.28.243.126:35099
--webserver_port=43831
--webserver_interface=127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:40:12 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315212, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:12.938274 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:12.891297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:35099 (local address 127.28.243.65:35721)
0809 22:40:12.902149 (+ 10852us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:12.902208 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:12.904992 (+ 2784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:12.909276 (+ 4284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:12.909289 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:12.910084 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:12.911000 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:12.911033 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:12.917592 (+ 6559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:12.917600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:12.919133 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:12.919155 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:12.921416 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:12.921426 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:12.921767 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:12.922716 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:12.922759 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:12.926256 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:12.931744 (+ 5488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:12.931756 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:12.931776 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:12.932683 (+ 907us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:12.933364 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:12.933370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:12.933374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:12.933674 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:12.935187 (+ 1513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:12.935197 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:12.935728 (+ 531us) client_negotiation.cc:769] Sending connection context
0809 22:40:12.936085 (+ 357us) client_negotiation.cc:240] Negotiation successful
0809 22:40:12.936812 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10419,"thread_start_us":14008,"threads_started":1}
I20260809 22:40:12.941037 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:12.890923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:35721 (local address 127.28.243.126:35099)
0809 22:40:12.895293 (+ 4370us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:12.895323 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:12.902423 (+ 7100us) server_negotiation.cc:407] Connection header received
0809 22:40:12.907076 (+ 4653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:12.907102 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:12.907738 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:12.908199 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:12.911267 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:12.913577 (+ 2310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:12.919378 (+ 5801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:12.920661 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:12.920933 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:12.926679 (+ 5746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:12.926693 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:12.926716 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:12.926782 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:12.931491 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:12.932916 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:12.932928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:12.932937 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:12.933185 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:12.933838 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:12.933844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:12.933848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:12.934861 (+ 1013us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:12.935081 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:12.939099 (+ 4018us) server_negotiation.cc:299] Negotiation successful
0809 22:40:12.939595 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3974,"thread_start_us":5949,"threads_started":1}
I20260809 22:40:12.945726 11224 heartbeater.cc:347] Connected to a master server at 127.28.243.126:35099
I20260809 22:40:12.946122 11224 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:12.946910 11224 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
W20260809 22:40:12.950651 11224 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:35099 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:40:12.950904 11224 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:12.951205 11224 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
I20260809 22:40:12.952759 11224 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:12.953055 11224 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
W20260809 22:40:12.954341 11224 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 22:40:13.433933 11230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:13.434371 11230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:13.434590 11230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:13.444180 11230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:13.444370 11230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:13.444653 11230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:40:13.456861 11230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:35885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=43831
--tserver_master_addrs=127.28.243.126:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:13.458513 11230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:13.460058 11230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:13.480576 11248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:13.480726 11249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:13.486148 11251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:13.487376 11230 server_base.cc:1034] running on GCE node
I20260809 22:40:13.488337 11230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:13.489949 11230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:13.491394 11230 hybrid_clock.cc:648] HybridClock initialized: now 1786315213491307 us; error 69 us; skew 500 ppm
I20260809 22:40:13.491796 11230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:13 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:13.495960 11230 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:40:13.498142 11230 webserver.cc:460] Webserver started at http://127.28.243.66:43831/ using document root <none> and password file <none>
I20260809 22:40:13.499148 11230 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:13.499343 11230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 22:40:13 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786315213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 09 22:40:13 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:13.547541 11230 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.006s sys 0.000s
I20260809 22:40:13.557029 11259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:13.559226 11230 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.000s
I20260809 22:40:13.559533 11230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "d076022427094727b54165074ea5ddc7"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "ff32a450655673df17087cc7919d00e5"
server_key_iv: "ed80f873ba928f4a3995f02f904a3f3b"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:13.561445 11230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:13.577389 11230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:13.582424 11230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:13.582756 11230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:13.584077 11230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:13.585731 11230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:13.585861 11230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:13.586011 11230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:13.586112 11230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:13.636132 11230 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:35885
I20260809 22:40:13.636089 11381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:35885 every 8 connection(s)
I20260809 22:40:13.638056 11230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:40:13.639719 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 11230
I20260809 22:40:13.640928 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.243.84:58207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:46247
--local_ip_for_outbound_sockets=127.28.243.67
--tserver_master_addrs=127.28.243.126:35099
--webserver_port=36615
--webserver_interface=127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:40:13 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:13.695905 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:13.655036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:52255 (local address 127.28.243.126:35099)
0809 22:40:13.665193 (+ 10157us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:13.665204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:13.668668 (+ 3464us) server_negotiation.cc:407] Connection header received
0809 22:40:13.673103 (+ 4435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:13.673111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:13.673220 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:13.673361 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:13.676674 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:13.677965 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:13.680971 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:13.681806 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:13.681939 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:13.687559 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:13.687570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:13.687579 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:13.687639 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:13.691762 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:13.692889 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:13.692896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:13.692900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:13.693057 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:13.693647 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:13.693653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:13.693656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:13.694053 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:13.694205 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:13.695236 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0809 22:40:13.695508 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9920,"thread_start_us":17373,"threads_started":1}
I20260809 22:40:13.698545 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:13.655692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:35099 (local address 127.28.243.66:52255)
0809 22:40:13.665967 (+ 10275us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:13.666031 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:13.668492 (+ 2461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:13.673707 (+ 5215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:13.673725 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:13.674976 (+ 1251us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:13.676347 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:13.676383 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:13.678204 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:13.678213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:13.680732 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:13.680755 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:13.682396 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:13.682407 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:13.682822 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:13.683936 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:13.683977 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:13.687163 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:13.691982 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:13.691994 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:13.692016 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:13.692731 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:13.693208 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:13.693213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:13.693217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:13.693513 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:13.694307 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:13.694316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:13.694771 (+ 455us) client_negotiation.cc:769] Sending connection context
0809 22:40:13.696399 (+ 1628us) client_negotiation.cc:240] Negotiation successful
0809 22:40:13.697120 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9914,"thread_start_us":9649,"threads_started":1}
I20260809 22:40:13.700603 11382 heartbeater.cc:347] Connected to a master server at 127.28.243.126:35099
I20260809 22:40:13.700977 11382 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:13.701766 11382 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
I20260809 22:40:13.703799 11014 ts_manager.cc:194] Registered new tserver with Master: d076022427094727b54165074ea5ddc7 (127.28.243.66:35885)
I20260809 22:40:13.705454 11014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:52255
I20260809 22:40:13.954524 11224 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:13.955021 11224 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
I20260809 22:40:13.956431 11014 ts_manager.cc:194] Registered new tserver with Master: a96c4f201d55423c98cb307a5bb38f83 (127.28.243.65:38441)
I20260809 22:40:13.957667 11014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:35721
Aug 09 22:40:14 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
W20260809 22:40:14.156646 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:14.157097 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:14.157315 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:14.169200 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:14.169368 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:14.169610 11390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:40:14.184026 11390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.243.84:58207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:46247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=36615
--tserver_master_addrs=127.28.243.126:35099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:14.185783 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:14.187531 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:14.206586 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:14.207036 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:14.214334 11399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:14.215073 11390 server_base.cc:1034] running on GCE node
I20260809 22:40:14.215950 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:14.217307 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:14.218590 11390 hybrid_clock.cc:648] HybridClock initialized: now 1786315214218520 us; error 52 us; skew 500 ppm
I20260809 22:40:14.219027 11390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:14 dist-test-slave-rgb1 krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:14.225296 11390 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:40:14.228960 11390 webserver.cc:460] Webserver started at http://127.28.243.67:36615/ using document root <none> and password file <none>
I20260809 22:40:14.230067 11390 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:14.230310 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 22:40:14 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:14.270402 11390 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.008s sys 0.003s
I20260809 22:40:14.274986 11407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:14.277256 11390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 22:40:14.277598 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "11626be3728244f6a3c3143d41dddaff"
format_stamp: "Formatted at 2026-08-09 22:40:10 on dist-test-slave-rgb1"
server_key: "2e4895f503f2234c6f5b10e206d9da0c"
server_key_iv: "d7de25b429f321b2f51460afea34abf5"
server_key_version: "kuduclusterkey@0"
I20260809 22:40:14.279136 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 09 22:40:14 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315213, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for HTTP/127.28.243.84@KRBTEST.COM
I20260809 22:40:14.322731 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:14.331223 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:14.331676 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:14.334249 11390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:14.336284 11390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:14.336421 11390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:14.336537 11390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:14.336642 11390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 22:40:14.480654 11390 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:46247
I20260809 22:40:14.480734 11530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:46247 every 8 connection(s)
I20260809 22:40:14.482731 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:40:14.488510 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 11390
Aug 09 22:40:14 dist-test-slave-rgb1 krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315214, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:14.537690 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.493054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.28.243.126:35099)
0809 22:40:14.516930 (+ 23876us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.516936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.516955 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:40:14.517005 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.517010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.517086 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.517189 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:14.519598 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.520844 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.522782 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.523529 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.523672 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.528661 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:14.528667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:14.528673 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:14.528729 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:14.533057 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.534050 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.534056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.534060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.534240 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.534735 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.534740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.534744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.535149 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:14.535446 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.537031 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.537285 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23738,"thread_start_us":23599,"threads_started":1}
I20260809 22:40:14.545094 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.486022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:52671 (local address 127.28.243.126:35099)
0809 22:40:14.493164 (+ 7142us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.493176 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.511740 (+ 18564us) server_negotiation.cc:407] Connection header received
0809 22:40:14.514638 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.514645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.514792 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.514932 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:14.526145 (+ 11213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.527322 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.529798 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.530543 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.530647 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.537332 (+ 6685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:14.537338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:14.537343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:14.537395 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:14.541040 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.541971 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.541977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.541980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.542222 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.542859 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.542864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.542868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.543280 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:14.543425 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.544388 (+ 963us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.544691 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6943,"thread_start_us":6757,"threads_started":1}
I20260809 22:40:14.546312 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.486699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:35099 (local address 127.28.243.67:52671)
0809 22:40:14.511494 (+ 24795us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:14.511577 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:14.514163 (+ 2586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:14.523164 (+ 9001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:14.523178 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:14.523924 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:14.525871 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.525903 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.527569 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.527574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.529617 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.529634 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.531064 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.531071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.531376 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.534014 (+ 2638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:14.534055 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:14.537016 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:14.541211 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:14.541221 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:14.541241 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:14.541823 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:14.542388 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:14.542392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:14.542396 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:14.542706 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:14.543449 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:14.543458 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:14.543925 (+ 467us) client_negotiation.cc:769] Sending connection context
0809 22:40:14.544243 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 22:40:14.544871 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24384,"thread_start_us":28646,"threads_started":1}
I20260809 22:40:14.547976 11531 heartbeater.cc:347] Connected to a master server at 127.28.243.126:35099
I20260809 22:40:14.548300 11531 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:14.549252 11531 heartbeater.cc:511] Master 127.28.243.126:35099 requested a full tablet report, sending...
I20260809 22:40:14.550972 11014 ts_manager.cc:194] Registered new tserver with Master: 11626be3728244f6a3c3143d41dddaff (127.28.243.67:46247)
I20260809 22:40:14.552242 11014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:52671
I20260809 22:40:14.557192 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:40:14.586802 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.563119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48582 (local address 127.28.243.126:35099)
0809 22:40:14.563319 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.563326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.567884 (+ 4558us) server_negotiation.cc:407] Connection header received
0809 22:40:14.568144 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.568150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.568272 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.568387 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:14.570833 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.571952 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.574180 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.574879 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.575022 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.579029 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:14.579037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:14.579042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:14.579093 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:14.582961 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.583974 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.583980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.583984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.584172 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.584740 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.584746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.584749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.585144 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:14.585403 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.585934 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.586196 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:40:14.593405 11014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:40:14.604832 11014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:40:14.668521 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.634892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:46247 (local address 127.0.0.1:37976)
0809 22:40:14.649738 (+ 14846us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:14.649783 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:14.650022 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:14.657405 (+ 7383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:14.657413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:14.657463 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:14.657983 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.658009 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.663755 (+ 5746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.663762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.665772 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.665786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.667832 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.667839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.668020 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.668054 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 22:40:14.668116 (+ 62us) client_negotiation.cc:240] Negotiation successful
0809 22:40:14.668228 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14551,"thread_start_us":14367,"threads_started":1}
I20260809 22:40:14.669806 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.635717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.28.243.67:46247)
0809 22:40:14.653160 (+ 17443us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.653173 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.653215 (+ 42us) server_negotiation.cc:407] Connection header received
0809 22:40:14.653334 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.653345 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.653618 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.658372 (+ 4754us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:14.658795 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.660983 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.666020 (+ 5037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.667567 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.668863 (+ 1296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.669152 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.669250 (+ 98us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.669432 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17197,"thread_start_us":17004,"threads_started":1}
I20260809 22:40:14.670156 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.636264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:35885 (local address 127.0.0.1:51136)
0809 22:40:14.651164 (+ 14900us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:14.651190 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:14.651352 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:14.653942 (+ 2590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:14.653954 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:14.654015 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:14.654555 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.654576 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.661305 (+ 6729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.661314 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.663471 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.663489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.667344 (+ 3855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.667350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.667586 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.667665 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 22:40:14.667885 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 22:40:14.668116 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14739,"thread_start_us":15476,"threads_started":1}
I20260809 22:40:14.670632 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.636758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.28.243.66:35885)
0809 22:40:14.650650 (+ 13892us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.650661 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.651503 (+ 842us) server_negotiation.cc:407] Connection header received
0809 22:40:14.651647 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.651660 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.652009 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.652251 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:14.654838 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.657093 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.663671 (+ 6578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.666753 (+ 3082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.666901 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.667161 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.668622 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.669444 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13721,"thread_start_us":15218,"threads_started":1}
I20260809 22:40:14.672834 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.650597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53862)
0809 22:40:14.661646 (+ 11049us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:14.661670 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:14.662371 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:14.663314 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:14.663318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:14.663343 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:14.663744 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.663756 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.665699 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.665703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.668123 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.668138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.672032 (+ 3894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.672037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.672163 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.672183 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 22:40:14.672301 (+ 118us) client_negotiation.cc:240] Negotiation successful
0809 22:40:14.672452 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10912,"thread_start_us":10719,"threads_started":1}
I20260809 22:40:14.672899 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.650646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.28.243.65:38441)
0809 22:40:14.661138 (+ 10492us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.661147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.661770 (+ 623us) server_negotiation.cc:407] Connection header received
0809 22:40:14.662586 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.662595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.662847 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.663046 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:14.663971 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.665500 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.668347 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.669740 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.669833 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.671475 (+ 1642us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.672360 (+ 885us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.672536 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10326,"thread_start_us":9925,"threads_started":1}
I20260809 22:40:14.674338 11460 tablet_service.cc:1467] Processing CreateTablet for tablet 19025701afd64b7da2b700b140d9484f (DEFAULT_TABLE table=test-table [id=bbd52f45b84e4ce4b97642f338d8cc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:14.674832 11313 tablet_service.cc:1467] Processing CreateTablet for tablet 19025701afd64b7da2b700b140d9484f (DEFAULT_TABLE table=test-table [id=bbd52f45b84e4ce4b97642f338d8cc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:14.676126 11460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19025701afd64b7da2b700b140d9484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:14.676266 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19025701afd64b7da2b700b140d9484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:14.676704 11156 tablet_service.cc:1467] Processing CreateTablet for tablet 19025701afd64b7da2b700b140d9484f (DEFAULT_TABLE table=test-table [id=bbd52f45b84e4ce4b97642f338d8cc89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:14.677984 11156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19025701afd64b7da2b700b140d9484f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:14.699564 11560 tablet_bootstrap.cc:492] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Bootstrap starting.
I20260809 22:40:14.701328 11561 tablet_bootstrap.cc:492] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Bootstrap starting.
I20260809 22:40:14.702553 11559 tablet_bootstrap.cc:492] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: Bootstrap starting.
I20260809 22:40:14.705569 11560 tablet_bootstrap.cc:654] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:14.705641 11561 tablet_bootstrap.cc:654] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:14.706459 11559 tablet_bootstrap.cc:654] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:14.706923 11560 log.cc:824] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:14.707085 11561 log.cc:824] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:14.707964 11559 log.cc:824] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:14.710341 11382 heartbeater.cc:503] Master 127.28.243.126:35099 was elected leader, sending a full tablet report...
I20260809 22:40:14.718129 11559 tablet_bootstrap.cc:492] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: No bootstrap required, opened a new log
I20260809 22:40:14.718572 11559 ts_tablet_manager.cc:1397] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260809 22:40:14.726820 11561 tablet_bootstrap.cc:492] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: No bootstrap required, opened a new log
I20260809 22:40:14.726955 11560 tablet_bootstrap.cc:492] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: No bootstrap required, opened a new log
I20260809 22:40:14.727164 11561 ts_tablet_manager.cc:1397] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.010s
I20260809 22:40:14.727244 11560 ts_tablet_manager.cc:1397] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.009s
I20260809 22:40:14.733137 11559 raft_consensus.cc:348] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.733660 11559 raft_consensus.cc:374] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:14.733767 11559 raft_consensus.cc:729] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a96c4f201d55423c98cb307a5bb38f83, State: Initialized, Role: FOLLOWER
I20260809 22:40:14.734443 11559 consensus_queue.cc:260] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.737848 11561 raft_consensus.cc:348] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.738317 11561 raft_consensus.cc:374] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:14.738427 11561 raft_consensus.cc:729] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11626be3728244f6a3c3143d41dddaff, State: Initialized, Role: FOLLOWER
I20260809 22:40:14.739156 11561 consensus_queue.cc:260] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.739447 11560 raft_consensus.cc:348] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.740016 11560 raft_consensus.cc:374] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:14.740159 11560 raft_consensus.cc:729] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d076022427094727b54165074ea5ddc7, State: Initialized, Role: FOLLOWER
I20260809 22:40:14.740896 11560 consensus_queue.cc:260] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.747198 11224 heartbeater.cc:503] Master 127.28.243.126:35099 was elected leader, sending a full tablet report...
I20260809 22:40:14.748275 11559 ts_tablet_manager.cc:1428] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: Time spent starting tablet: real 0.030s user 0.021s sys 0.004s
I20260809 22:40:14.755770 11531 heartbeater.cc:503] Master 127.28.243.126:35099 was elected leader, sending a full tablet report...
I20260809 22:40:14.756629 11561 ts_tablet_manager.cc:1428] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Time spent starting tablet: real 0.029s user 0.019s sys 0.009s
I20260809 22:40:14.758699 11560 ts_tablet_manager.cc:1428] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Time spent starting tablet: real 0.031s user 0.017s sys 0.014s
I20260809 22:40:14.783870 11568 raft_consensus.cc:482] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:40:14.784225 11568 raft_consensus.cc:504] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.786432 11568 leader_election.cc:288] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11626be3728244f6a3c3143d41dddaff (127.28.243.67:46247), a96c4f201d55423c98cb307a5bb38f83 (127.28.243.65:38441)
I20260809 22:40:14.794485 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.787134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:47455 (local address 127.28.243.67:46247)
0809 22:40:14.787384 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.787393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.788059 (+ 666us) server_negotiation.cc:407] Connection header received
0809 22:40:14.788297 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.788306 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.788416 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.788508 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:14.789475 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.790192 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.792520 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.793350 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.793560 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.793647 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.794087 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.794234 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 22:40:14.794606 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.786897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:46247 (local address 127.28.243.66:47455)
0809 22:40:14.787926 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:14.787948 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:14.788098 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:14.788832 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:14.788840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:14.788873 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:14.789316 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.789329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.790331 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.790338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.792351 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.792368 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.793601 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.793608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.793887 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.793923 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 22:40:14.794028 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 22:40:14.794179 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":110,"thread_start_us":672,"threads_started":1}
I20260809 22:40:14.795703 11482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19025701afd64b7da2b700b140d9484f" candidate_uuid: "d076022427094727b54165074ea5ddc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11626be3728244f6a3c3143d41dddaff" is_pre_election: true
I20260809 22:40:14.796056 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.788451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:42579 (local address 127.28.243.65:38441)
0809 22:40:14.788678 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.788684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.789319 (+ 635us) server_negotiation.cc:407] Connection header received
0809 22:40:14.789516 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.789521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.789592 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.789651 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:14.790430 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.791526 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.793886 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.795123 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.795300 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.795373 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.795663 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.795793 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 22:40:14.796166 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.788111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.28.243.66:42579)
0809 22:40:14.789189 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:14.789209 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:14.789352 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:14.789834 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:14.789838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:14.789861 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:14.790218 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.790228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.791693 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.791698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.793689 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:14.793704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.795323 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.795330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:14.795496 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.795520 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 22:40:14.795626 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 22:40:14.795759 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":28,"thread_start_us":724,"threads_started":1}
I20260809 22:40:14.796305 11482 raft_consensus.cc:2454] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d076022427094727b54165074ea5ddc7 in term 0.
I20260809 22:40:14.797163 11262 leader_election.cc:302] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11626be3728244f6a3c3143d41dddaff, d076022427094727b54165074ea5ddc7; no voters:
I20260809 22:40:14.797142 11177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19025701afd64b7da2b700b140d9484f" candidate_uuid: "d076022427094727b54165074ea5ddc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a96c4f201d55423c98cb307a5bb38f83" is_pre_election: true
I20260809 22:40:14.797608 11568 raft_consensus.cc:2781] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:40:14.797659 11177 raft_consensus.cc:2454] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d076022427094727b54165074ea5ddc7 in term 0.
I20260809 22:40:14.797788 11568 raft_consensus.cc:482] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:40:14.797897 11568 raft_consensus.cc:3037] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:14.800817 11568 raft_consensus.cc:504] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.801524 11568 leader_election.cc:288] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [CANDIDATE]: Term 1 election: Requested vote from peers 11626be3728244f6a3c3143d41dddaff (127.28.243.67:46247), a96c4f201d55423c98cb307a5bb38f83 (127.28.243.65:38441)
I20260809 22:40:14.802084 11482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19025701afd64b7da2b700b140d9484f" candidate_uuid: "d076022427094727b54165074ea5ddc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11626be3728244f6a3c3143d41dddaff"
I20260809 22:40:14.802211 11177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19025701afd64b7da2b700b140d9484f" candidate_uuid: "d076022427094727b54165074ea5ddc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a96c4f201d55423c98cb307a5bb38f83"
I20260809 22:40:14.802325 11482 raft_consensus.cc:3037] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:14.802436 11177 raft_consensus.cc:3037] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:14.804823 11482 raft_consensus.cc:2454] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d076022427094727b54165074ea5ddc7 in term 1.
I20260809 22:40:14.805330 11262 leader_election.cc:302] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11626be3728244f6a3c3143d41dddaff, d076022427094727b54165074ea5ddc7; no voters:
I20260809 22:40:14.805658 11568 raft_consensus.cc:2781] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:14.805696 11177 raft_consensus.cc:2454] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d076022427094727b54165074ea5ddc7 in term 1.
I20260809 22:40:14.806423 11568 raft_consensus.cc:686] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 1 LEADER]: Becoming Leader. State: Replica: d076022427094727b54165074ea5ddc7, State: Running, Role: LEADER
I20260809 22:40:14.806996 11568 consensus_queue.cc:237] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } }
I20260809 22:40:14.812446 11013 catalog_manager.cc:5495] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d076022427094727b54165074ea5ddc7 (127.28.243.66). New cstate: current_term: 1 leader_uuid: "d076022427094727b54165074ea5ddc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d076022427094727b54165074ea5ddc7" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 35885 } health_report { overall_health: HEALTHY } } }
I20260809 22:40:14.834362 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.825262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.28.243.66:35885)
0809 22:40:14.825477 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.825485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.825675 (+ 190us) server_negotiation.cc:407] Connection header received
0809 22:40:14.826014 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.826021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.826133 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.826268 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:40:14.826930 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.828116 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.830317 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.831130 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.831256 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.832834 (+ 1578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:40:14.833607 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:40:14.833761 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.833967 (+ 206us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.834075 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 22:40:14.844898 11177 raft_consensus.cc:1261] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 1 FOLLOWER]: Refusing update from remote peer d076022427094727b54165074ea5ddc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:14.844918 11482 raft_consensus.cc:1261] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 1 FOLLOWER]: Refusing update from remote peer d076022427094727b54165074ea5ddc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:14.846005 11572 consensus_queue.cc:1035] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a96c4f201d55423c98cb307a5bb38f83" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 38441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:14.846518 11568 consensus_queue.cc:1035] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11626be3728244f6a3c3143d41dddaff" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:14.886821 11014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:14.887171 11014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:14.891985 11014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4771d126e81f45c5b234466b78fc897f: Sending DeleteTablet for 3 replicas of tablet 19025701afd64b7da2b700b140d9484f
I20260809 22:40:14.893330 11460 tablet_service.cc:1514] Processing DeleteTablet for tablet 19025701afd64b7da2b700b140d9484f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:14 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:37976
I20260809 22:40:14.895215 11156 tablet_service.cc:1514] Processing DeleteTablet for tablet 19025701afd64b7da2b700b140d9484f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:14 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:53862
I20260809 22:40:14.896028 11313 tablet_service.cc:1514] Processing DeleteTablet for tablet 19025701afd64b7da2b700b140d9484f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:14 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:51136
I20260809 22:40:14.907313 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 11053
I20260809 22:40:14.908452 11585 tablet_replica.cc:330] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: stopping tablet replica
I20260809 22:40:14.909279 11585 raft_consensus.cc:2229] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:14.910064 11585 raft_consensus.cc:2258] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:14.911006 11586 tablet_replica.cc:330] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: stopping tablet replica
I20260809 22:40:14.912935 11586 raft_consensus.cc:2229] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:14.913733 11585 ts_tablet_manager.cc:1905] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:14.914319 11586 raft_consensus.cc:2258] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:14.922065 11585 ts_tablet_manager.cc:1918] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:14.922353 11585 log.cc:1196] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/19025701afd64b7da2b700b140d9484f
I20260809 22:40:14.923105 11585 ts_tablet_manager.cc:1939] T 19025701afd64b7da2b700b140d9484f P 11626be3728244f6a3c3143d41dddaff: Deleting consensus metadata
I20260809 22:40:14.925374 10998 catalog_manager.cc:4841] TS 11626be3728244f6a3c3143d41dddaff (127.28.243.67:46247): tablet 19025701afd64b7da2b700b140d9484f (table test-table [id=bbd52f45b84e4ce4b97642f338d8cc89]) successfully deleted
I20260809 22:40:14.932152 11586 ts_tablet_manager.cc:1905] T 19025701afd64b7da2b700b140d9484f P a96c4f201d55423c98cb307a5bb38f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:14.936223 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:14.909602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.28.243.65:38441)
0809 22:40:14.909795 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:14.909803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:14.909826 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:40:14.909930 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:14.909936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:14.910031 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:14.910166 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:14.913590 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.914933 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.918698 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:14.919416 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:14.919567 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:14.927242 (+ 7675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:14.927262 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:14.927283 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:14.927334 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:14.931604 (+ 4270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.932911 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.932921 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.932925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.933125 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:14.933640 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:14.933646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:14.933650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:14.934166 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:14.934525 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:14.935494 (+ 969us) server_negotiation.cc:299] Negotiation successful
0809 22:40:14.935809 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:40:14.938174 11219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:14.945878 11587 tablet_replica.cc:330] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: stopping tablet replica
I20260809 22:40:14.946460 11587 raft_consensus.cc:2229] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:40:14.947266 11587 raft_consensus.cc:2258] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:14.949877 11587 ts_tablet_manager.cc:1905] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:14.959095 11587 ts_tablet_manager.cc:1918] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:14.959317 11587 log.cc:1196] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/19025701afd64b7da2b700b140d9484f
I20260809 22:40:14.959939 11587 ts_tablet_manager.cc:1939] T 19025701afd64b7da2b700b140d9484f P d076022427094727b54165074ea5ddc7: Deleting consensus metadata
I20260809 22:40:14.963169 10997 catalog_manager.cc:4841] TS d076022427094727b54165074ea5ddc7 (127.28.243.66:35885): tablet 19025701afd64b7da2b700b140d9484f (table test-table [id=bbd52f45b84e4ce4b97642f338d8cc89]) successfully deleted
I20260809 22:40:15.057837 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 11053
W20260809 22:40:15.079523 10998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.65:38441 (error 108)
I20260809 22:40:15.079679 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 11230
I20260809 22:40:15.081939 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.081501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53884)
0809 22:40:15.081673 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:15.081801 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 22:40:15.082224 10998 catalog_manager.cc:4584] TS a96c4f201d55423c98cb307a5bb38f83 (127.28.243.65:38441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19025701afd64b7da2b700b140d9484f: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
I20260809 22:40:15.098624 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.081131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.28.243.66:35885)
0809 22:40:15.081305 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:15.081310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:15.081327 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:40:15.081381 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:15.081385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:15.081473 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:15.081606 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:15.083734 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:15.084789 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:15.087260 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:15.087840 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:15.087976 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:15.091689 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:15.091706 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:15.091723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:15.091770 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:15.095054 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:15.096099 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:15.096112 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:15.096115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:15.096294 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:15.096803 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:15.096817 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:15.096819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:15.097301 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:15.097562 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:15.098014 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 22:40:15.098220 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 22:40:15.099350 11377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:15.136982 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.136039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53892)
0809 22:40:15.136473 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:15.136787 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
I20260809 22:40:15.138638 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.137951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53896)
0809 22:40:15.138336 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:15.138481 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 22:40:15.199326 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.198801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53900)
0809 22:40:15.199031 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:15.199144 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 22:40:15.200703 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.200198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53912)
0809 22:40:15.200422 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:15.200501 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 22:40:15.223030 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 11230
I20260809 22:40:15.243958 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 11390
I20260809 22:40:15.260886 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.244978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.28.243.67:46247)
0809 22:40:15.245193 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:15.245201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:15.245365 (+ 164us) server_negotiation.cc:407] Connection header received
0809 22:40:15.245501 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:15.245505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:15.245572 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:15.245682 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:15.247750 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:15.249151 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:15.250721 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:15.251177 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:15.251280 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:15.254205 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:15.254221 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:15.254238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:15.254282 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:15.257616 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:15.258388 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:15.258396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:15.258398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:15.258536 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:15.258996 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:15.259001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:15.259003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:15.259500 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:15.259781 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:15.260109 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 22:40:15.260304 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:40:15.262135 11525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:15.281582 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.281043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53924)
0809 22:40:15.281275 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:15.281371 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260809 22:40:15.282877 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:15.282450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:38441 (local address 127.0.0.1:53926)
0809 22:40:15.282686 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:15.282767 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260809 22:40:15.355374 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 11390
I20260809 22:40:15.374703 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 10976
I20260809 22:40:15.376116 11046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:15.500639 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 10976
I20260809 22:40:15.531735 29645 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 22:40:15.878420 29645 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 22:40:15.878674 29645 mini_ranger.cc:66] Stopping Ranger...
I20260809 22:40:16.236037 29645 mini_ranger.cc:68] Stopped Ranger
2026-08-09 22:40:16.236 UTC [9978] LOG: received smart shutdown request
2026-08-09 22:40:16.240 UTC [9978] LOG: background worker "logical replication launcher" (PID 9987) exited with exit code 1
2026-08-09 22:40:16.240 UTC [9982] LOG: shutting down
2026-08-09 22:40:16.483 UTC [9978] LOG: database system is shut down
2026-08-09T22:40:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58357 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:40:16 dist-test-slave-rgb1 krb5kdc[11602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:40:16 dist-test-slave-rgb1 krb5kdc[11602](info): set up 2 sockets
Aug 09 22:40:16 dist-test-slave-rgb1 krb5kdc[11602](info): commencing operation
krb5kdc: starting...
W20260809 22:40:17.834182 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:40:18 dist-test-slave-rgb1 krb5kdc[11602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:40:18.149914 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39017
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39017,127.28.243.125:39113,127.28.243.124:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:18.508230 11616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:18.508713 11616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:18.508910 11616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:18.517477 11616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:18.517612 11616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:18.517693 11616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:18.517791 11616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:18.517881 11616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:18.529436 11616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39017,127.28.243.125:39113,127.28.243.124:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:18.530975 11616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:18.532485 11616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:18.547273 11625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:18.547323 11622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:18.547294 11623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:18.548147 11616 server_base.cc:1034] running on GCE node
I20260809 22:40:18.549156 11616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:18.550750 11616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:18.551985 11616 hybrid_clock.cc:648] HybridClock initialized: now 1786315218551935 us; error 43 us; skew 500 ppm
I20260809 22:40:18.552457 11616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:18 dist-test-slave-rgb1 krb5kdc[11602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:18.556222 11616 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:18.558344 11616 webserver.cc:460] Webserver started at http://127.28.243.126:41597/ using document root <none> and password file <none>
I20260809 22:40:18.559199 11616 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:18.559326 11616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:18.559739 11616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:18.563287 11616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "583fb8386b7240869326fadc3759f08b"
format_stamp: "Formatted at 2026-08-09 22:40:18 on dist-test-slave-rgb1"
server_key: "76fd2abe0978b7f2af237ef2890ba993"
server_key_iv: "3e697380cd08da3a5446d60d845a1016"
server_key_version: "encryptionkey@0"
I20260809 22:40:18.564248 11616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "583fb8386b7240869326fadc3759f08b"
format_stamp: "Formatted at 2026-08-09 22:40:18 on dist-test-slave-rgb1"
server_key: "76fd2abe0978b7f2af237ef2890ba993"
server_key_iv: "3e697380cd08da3a5446d60d845a1016"
server_key_version: "encryptionkey@0"
I20260809 22:40:18.570375 11616 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260809 22:40:18.575112 11633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:18.577028 11616 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 22:40:18.577235 11616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "583fb8386b7240869326fadc3759f08b"
format_stamp: "Formatted at 2026-08-09 22:40:18 on dist-test-slave-rgb1"
server_key: "76fd2abe0978b7f2af237ef2890ba993"
server_key_iv: "3e697380cd08da3a5446d60d845a1016"
server_key_version: "encryptionkey@0"
I20260809 22:40:18.577491 11616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:18.599145 11616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:18.604611 11616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:18.604961 11616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:18.627981 11616 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:39017
I20260809 22:40:18.628103 11686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:39017 every 8 connection(s)
I20260809 22:40:18.629611 11616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:18.631213 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 11616
I20260809 22:40:18.631464 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:40:18.632334 29645 external_mini_cluster.cc:1428] Setting key 5cd7009423529dd8850954d8a32183b9
I20260809 22:40:18.634902 11687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:18.645514 11687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.243.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.125@KRBTEST.COM" created.
Aug 09 22:40:18 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:18.674675 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.648941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39017 (local address 127.0.0.1:51668)
0809 22:40:18.649939 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.650002 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:18.653076 (+ 3074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:18.654210 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:18.654220 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:18.655119 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:18.655692 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:18.655730 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:18.657568 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:18.657572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:18.658700 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:18.658712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:18.659772 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:18.659779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:18.659960 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:18.660742 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:18.660766 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:18.666642 (+ 5876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:18.670264 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:18.670278 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:18.670292 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:18.670730 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:18.671170 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:18.671177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:18.671180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:18.671349 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:18.672204 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:18.672217 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:18.672453 (+ 236us) client_negotiation.cc:769] Sending connection context
0809 22:40:18.672769 (+ 316us) client_negotiation.cc:240] Negotiation successful
0809 22:40:18.673299 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":410,"threads_started":1}
I20260809 22:40:18.676609 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.648941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.28.243.126:39017)
0809 22:40:18.650952 (+ 2011us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:18.650958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:18.651008 (+ 50us) server_negotiation.cc:407] Connection header received
0809 22:40:18.653559 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:18.653570 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:18.653857 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:18.654105 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:18.656019 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:18.657400 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:18.658853 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:18.659504 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:18.659725 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:18.666923 (+ 7198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:18.666932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:18.666954 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:18.667001 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:18.670093 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:18.670875 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:18.670891 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:18.670893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:18.671055 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:18.671466 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:18.671473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:18.671477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:18.672058 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:18.672282 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:18.676009 (+ 3727us) server_negotiation.cc:299] Negotiation successful
0809 22:40:18.676236 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"spinlock_wait_cycles":49920,"thread_start_us":1538,"threads_started":1}
I20260809 22:40:18.679603 11687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:40:18.680913 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.680361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35036)
0809 22:40:18.680613 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.680748 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260809 22:40:18.681201 11634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
I20260809 22:40:18.683290 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.682806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35052)
0809 22:40:18.683007 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.683106 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 22:40:18.683629 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:39113: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab.
Entry for principal kudu/127.28.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab.
I20260809 22:40:18.702690 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:39113
--webserver_interface=127.28.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab
--principal=kudu/127.28.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39017,127.28.243.125:39113,127.28.243.124:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:18.733146 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } attempt: 1
I20260809 22:40:18.734505 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.733931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35066)
0809 22:40:18.734185 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.734320 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 22:40:18.735694 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.735217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35080)
0809 22:40:18.735411 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.735518 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 22:40:18.735981 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:39113: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
I20260809 22:40:18.804309 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } attempt: 2
I20260809 22:40:18.805852 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.805254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35082)
0809 22:40:18.805547 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.805664 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 22:40:18.807178 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.806706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35092)
0809 22:40:18.806888 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.807006 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 22:40:18.807439 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:39113: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
I20260809 22:40:18.898759 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } attempt: 3
I20260809 22:40:18.900207 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.899528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35108)
0809 22:40:18.899834 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.899991 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260809 22:40:18.901512 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:18.901000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35120)
0809 22:40:18.901210 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:18.901325 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 22:40:18.901808 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:39113: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
I20260809 22:40:19.045162 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } attempt: 4
I20260809 22:40:19.046573 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.045907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35122)
0809 22:40:19.046278 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.046392 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260809 22:40:19.047775 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.047321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35136)
0809 22:40:19.047499 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.047623 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260809 22:40:19.048048 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:39113: Network error: Client connection negotiation failed: client connection to 127.28.243.125:39113: connect: Connection refused (error 111)
W20260809 22:40:19.070153 11694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:19.070467 11694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:19.070564 11694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:19.078197 11694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:19.078325 11694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:19.078373 11694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:19.078440 11694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:19.078486 11694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:19.089128 11694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39017,127.28.243.125:39113,127.28.243.124:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab
--principal=kudu/127.28.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:39113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:19.090488 11694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:19.091914 11694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:19.104508 11701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:19.104537 11699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:19.104748 11703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:19.105577 11694 server_base.cc:1034] running on GCE node
I20260809 22:40:19.106449 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:19.108420 11694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:19.109745 11694 hybrid_clock.cc:648] HybridClock initialized: now 1786315219109674 us; error 126 us; skew 500 ppm
I20260809 22:40:19.110173 11694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:19.113802 11694 init.cc:377] Logged in from keytab as kudu/127.28.243.125@KRBTEST.COM (short username kudu)
I20260809 22:40:19.115628 11694 webserver.cc:460] Webserver started at http://127.28.243.125:39141/ using document root <none> and password file <none>
I20260809 22:40:19.116333 11694 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:19.116447 11694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:19.116803 11694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:19.120038 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data/instance:
uuid: "39b8748b026f4eed9eb2d3873fc5f564"
format_stamp: "Formatted at 2026-08-09 22:40:19 on dist-test-slave-rgb1"
server_key: "08f1a0292ee51284f34eb112b488e92a"
server_key_iv: "915cbee49ea456a8e6582c77070ef0d2"
server_key_version: "encryptionkey@0"
I20260809 22:40:19.120985 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/wal/instance:
uuid: "39b8748b026f4eed9eb2d3873fc5f564"
format_stamp: "Formatted at 2026-08-09 22:40:19 on dist-test-slave-rgb1"
server_key: "08f1a0292ee51284f34eb112b488e92a"
server_key_iv: "915cbee49ea456a8e6582c77070ef0d2"
server_key_version: "encryptionkey@0"
I20260809 22:40:19.126725 11694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260809 22:40:19.130952 11710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:19.132925 11694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:40:19.133131 11694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/wal
uuid: "39b8748b026f4eed9eb2d3873fc5f564"
format_stamp: "Formatted at 2026-08-09 22:40:19 on dist-test-slave-rgb1"
server_key: "08f1a0292ee51284f34eb112b488e92a"
server_key_iv: "915cbee49ea456a8e6582c77070ef0d2"
server_key_version: "encryptionkey@0"
I20260809 22:40:19.133337 11694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:19.157543 11694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:19.162976 11694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:19.163271 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:19.185103 11694 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.125:39113
I20260809 22:40:19.185135 11764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.125:39113 every 8 connection(s)
I20260809 22:40:19.186630 11694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
I20260809 22:40:19.190653 11765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:19.193758 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 11694
I20260809 22:40:19.193964 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-1/wal/instance
I20260809 22:40:19.194568 29645 external_mini_cluster.cc:1428] Setting key 22db8a0304cf38aed9649b389ea2c300
I20260809 22:40:19.200277 11765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.124@KRBTEST.COM; defaulting to no policy
I20260809 22:40:19.227020 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.204090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.28.243.126:39017)
0809 22:40:19.204717 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.204726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.206099 (+ 1373us) server_negotiation.cc:407] Connection header received
0809 22:40:19.208282 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.208292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.208395 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.208533 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.210463 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.211770 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.213709 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.214519 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.214693 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.219285 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.219290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.219293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.219380 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.222842 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.224120 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.224126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.224130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.224274 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.224865 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.224869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.224871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.225364 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.225497 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.226439 (+ 942us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.226649 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":297,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.124@KRBTEST.COM" created.
I20260809 22:40:19.228003 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.204887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39017 (local address 127.0.0.1:51674)
0809 22:40:19.205868 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.205931 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.207971 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.208885 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.208894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.209533 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.210149 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.210167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.212044 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.212049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.213533 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.213546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.214785 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.214790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.214988 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.215842 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.215871 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.218979 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.223037 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.223052 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.223078 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.223849 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.224417 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.224420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.224422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.224699 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.225541 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.225547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.225908 (+ 361us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.226226 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.226648 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":447,"threads_started":1}
I20260809 22:40:19.230376 11765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:40:19.248413 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.231130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35142)
0809 22:40:19.231391 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.231419 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.231538 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.232905 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.232909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.233380 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.233551 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.233560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.235615 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.235620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.236750 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.236764 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.238085 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.238090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.238294 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.239055 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.239083 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.241852 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.245524 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.245533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.245537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.245950 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.246399 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.246405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.246409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.246582 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.247268 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.247277 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.247454 (+ 177us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.247611 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.247917 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 22:40:19.248463 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.231437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35142 (local address 127.28.243.125:39113)
0809 22:40:19.232270 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.232281 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.232319 (+ 38us) server_negotiation.cc:407] Connection header received
0809 22:40:19.232389 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.232400 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.232701 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.232879 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.233737 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.235435 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.236911 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.237902 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.238047 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.242105 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.242118 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.242133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.242171 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.245363 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.246127 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.246142 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.246144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.246268 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.246713 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.246720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.246729 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.247127 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.247380 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.247789 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.248018 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":438,"threads_started":1}
I20260809 22:40:19.251395 11765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:40:19.252972 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.252315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42058)
0809 22:40:19.252528 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.252720 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260809 22:40:19.253324 11713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.255236 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.254830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42070)
0809 22:40:19.255026 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.255125 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 22:40:19.255549 11765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab.
Entry for principal kudu/127.28.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab.
I20260809 22:40:19.266211 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:34457
--webserver_interface=127.28.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab
--principal=kudu/127.28.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39017,127.28.243.125:39113,127.28.243.124:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:19.304904 11765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 1
I20260809 22:40:19.306249 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.305730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42084)
0809 22:40:19.305944 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.306090 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 22:40:19.307382 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.307003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42090)
0809 22:40:19.307197 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.307267 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 22:40:19.307610 11765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.347385 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } attempt: 5
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:40:19.364001 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.348119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35148)
0809 22:40:19.348391 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.348416 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.348649 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.349140 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.349148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.349679 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.349905 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.349917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.351015 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.351023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.352721 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.352742 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.353604 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.353611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.353830 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.354574 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.354598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.357102 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.361314 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.361318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.361322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.361762 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.362247 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.362251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.362253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.362388 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.363085 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.363090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.363253 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.363390 (+ 137us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.363676 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 22:40:19.364130 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.348311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.28.243.125:39113)
0809 22:40:19.348472 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.348477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.348514 (+ 37us) server_negotiation.cc:407] Connection header received
0809 22:40:19.348810 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.348814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.348956 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.349087 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.350128 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.350859 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.352928 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.353430 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.353541 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.357325 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.357332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.357336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.357380 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.361124 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.361927 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.361936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.361941 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.362119 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.362542 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.362548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.362553 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.362932 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.363090 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.363584 (+ 494us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.363822 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 22:40:19.365521 11687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:40:19.366861 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.366355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42106)
0809 22:40:19.366628 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.366710 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 22:40:19.368013 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.367643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42122)
0809 22:40:19.367801 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.367900 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 22:40:19.368224 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.375846 11765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 2
I20260809 22:40:19.377282 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.376690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42134)
0809 22:40:19.376896 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.377008 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 22:40:19.378473 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.378107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42140)
0809 22:40:19.378267 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.378350 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 22:40:19.378715 11765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.419478 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 1
I20260809 22:40:19.420840 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.420234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42152)
0809 22:40:19.420441 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.420544 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 22:40:19.421893 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.421555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42164)
0809 22:40:19.421708 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.421791 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 22:40:19.422106 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.470039 11765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 3
I20260809 22:40:19.471361 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.470836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42178)
0809 22:40:19.471107 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.471201 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260809 22:40:19.472841 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.472269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42192)
0809 22:40:19.472622 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.472720 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260809 22:40:19.473090 11765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.490347 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 2
I20260809 22:40:19.491681 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.491162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42204)
0809 22:40:19.491421 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.491512 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 22:40:19.492759 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.492344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42208)
0809 22:40:19.492505 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.492649 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 22:40:19.492964 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.599212 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 3
I20260809 22:40:19.600962 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.600300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42210)
0809 22:40:19.600551 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.600733 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260809 22:40:19.602252 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.601747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42216)
0809 22:40:19.602046 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.602132 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260809 22:40:19.602473 11687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
I20260809 22:40:19.616398 11765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 4
I20260809 22:40:19.617786 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.617249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42230)
0809 22:40:19.617523 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.617615 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 22:40:19.618860 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.618491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42244)
0809 22:40:19.618661 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.618746 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 22:40:19.619127 11765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:34457: Network error: Client connection negotiation failed: client connection to 127.28.243.124:34457: connect: Connection refused (error 111)
W20260809 22:40:19.633037 11774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:19.633373 11774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:19.633528 11774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:19.641291 11774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:19.641448 11774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:19.641549 11774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:19.641660 11774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:19.641738 11774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:19.652007 11774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39017,127.28.243.125:39113,127.28.243.124:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab
--principal=kudu/127.28.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:19.653450 11774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:19.654788 11774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:19.667640 11783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:19.667747 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:19.667722 11781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:19.668305 11774 server_base.cc:1034] running on GCE node
I20260809 22:40:19.669221 11774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:19.670670 11774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:19.671897 11774 hybrid_clock.cc:648] HybridClock initialized: now 1786315219671821 us; error 59 us; skew 500 ppm
I20260809 22:40:19.672327 11774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:19.676029 11774 init.cc:377] Logged in from keytab as kudu/127.28.243.124@KRBTEST.COM (short username kudu)
I20260809 22:40:19.677798 11774 webserver.cc:460] Webserver started at http://127.28.243.124:41543/ using document root <none> and password file <none>
I20260809 22:40:19.678700 11774 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:19.678862 11774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:19.679188 11774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:19.682757 11774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data/instance:
uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6"
format_stamp: "Formatted at 2026-08-09 22:40:19 on dist-test-slave-rgb1"
server_key: "e8f994b792e8a5218310be73ff22445e"
server_key_iv: "36b0e33fb34f0108e32e1c3f4eafb628"
server_key_version: "encryptionkey@0"
I20260809 22:40:19.683619 11774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/wal/instance:
uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6"
format_stamp: "Formatted at 2026-08-09 22:40:19 on dist-test-slave-rgb1"
server_key: "e8f994b792e8a5218310be73ff22445e"
server_key_iv: "36b0e33fb34f0108e32e1c3f4eafb628"
server_key_version: "encryptionkey@0"
I20260809 22:40:19.689901 11774 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 22:40:19.694103 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:19.695622 11774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 22:40:19.695833 11774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/wal
uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6"
format_stamp: "Formatted at 2026-08-09 22:40:19 on dist-test-slave-rgb1"
server_key: "e8f994b792e8a5218310be73ff22445e"
server_key_iv: "36b0e33fb34f0108e32e1c3f4eafb628"
server_key_version: "encryptionkey@0"
I20260809 22:40:19.696072 11774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:19.717166 11774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:19.722648 11774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:19.723069 11774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:19.745844 11774 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.124:34457
I20260809 22:40:19.745949 11843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.124:34457 every 8 connection(s)
I20260809 22:40:19.747321 11774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
I20260809 22:40:19.747925 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 11774
I20260809 22:40:19.748225 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786314994965430-29645-0/minicluster-data/master-2/wal/instance
I20260809 22:40:19.748950 29645 external_mini_cluster.cc:1428] Setting key c2d3be9db8c28f0ba93a9459d5086e74
I20260809 22:40:19.751878 11844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:19.759109 11844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:19.778154 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.750964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.28.243.124:34457)
0809 22:40:19.751833 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.751847 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.751896 (+ 49us) server_negotiation.cc:407] Connection header received
0809 22:40:19.753848 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.753871 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.754288 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.754743 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.756458 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.757997 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.759604 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.760481 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.760717 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.768014 (+ 7297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.768022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.768035 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.768076 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.772134 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.773447 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.773462 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.773473 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.773727 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.774317 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.774326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.774332 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.775038 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.775309 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.775868 (+ 559us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.776501 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":407,"threads_started":1}
I20260809 22:40:19.779744 11687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 4
I20260809 22:40:19.784819 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.761855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39017 (local address 127.0.0.1:51682)
0809 22:40:19.762792 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.762843 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.763121 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.766020 (+ 2899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.766079 (+ 59us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.767226 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.767520 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.767541 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.768826 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.768831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.770709 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.770726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.771580 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.771588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.771800 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.772979 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.773002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.775909 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.780856 (+ 4947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.780868 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.780880 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.781676 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.782232 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.782236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.782239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.782460 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.783430 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.783439 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.783662 (+ 223us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.783887 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.784283 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"spinlock_wait_cycles":25344,"thread_start_us":510,"threads_started":1}
I20260809 22:40:19.785650 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.762003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.28.243.126:39017)
0809 22:40:19.765423 (+ 3420us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.765430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.765467 (+ 37us) server_negotiation.cc:407] Connection header received
0809 22:40:19.765594 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.765602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.765710 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.765865 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.767714 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.768453 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.770888 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.771364 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.771513 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.776188 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.776196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.776201 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.776251 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.780664 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.781815 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.781821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.781825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.782044 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.782625 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.782630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.782633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.783214 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.783430 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.784827 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.785106 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3264,"thread_start_us":3099,"threads_started":1}
I20260809 22:40:19.787279 11844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:40:19.798512 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.780761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.28.243.124:34457)
0809 22:40:19.780983 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.780991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.781015 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:40:19.781198 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.781204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.781284 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.781399 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.782516 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.783738 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.785957 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.786666 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.786799 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.791349 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.791355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.791359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.791407 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.795050 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.796021 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.796028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.796031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.796214 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.796787 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.796793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.796797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.797242 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.797440 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.798014 (+ 574us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.798212 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 22:40:19.798659 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.780544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42262)
0809 22:40:19.780890 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.780911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.781045 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.781467 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.781472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.782021 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.782310 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.782329 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.783898 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.783904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.785743 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.785766 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.786846 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.786855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.787055 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.787801 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.787826 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.791159 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.795205 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.795212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.795216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.795839 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.796369 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.796378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.796385 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.796617 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.797395 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.797416 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.797609 (+ 193us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.797827 (+ 218us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.798151 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":3}
I20260809 22:40:19.805374 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b: Bootstrap starting.
I20260809 22:40:19.806396 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.782485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.28.243.126:39017)
0809 22:40:19.783611 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.783617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.783642 (+ 25us) server_negotiation.cc:407] Connection header received
0809 22:40:19.783696 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.783703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.783780 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.783954 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.786016 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.787425 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.789980 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.790685 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.790825 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.796649 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.796657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.796663 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.796707 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.802226 (+ 5519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.803308 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.803316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.803320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.803517 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.804004 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.804009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.804012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.804433 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.804795 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.805750 (+ 955us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.805991 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":971,"thread_start_us":700,"threads_started":1}
I20260809 22:40:19.807193 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.788205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.28.243.125:39113)
0809 22:40:19.788390 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.788397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.788444 (+ 47us) server_negotiation.cc:407] Connection header received
0809 22:40:19.788715 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.788722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.788806 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.788948 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.789717 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.790551 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.793704 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.794507 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.794679 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.799358 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.799367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.799374 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.799422 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.803688 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.804683 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.804690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.804694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.804891 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.805368 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.805374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.805378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.805913 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.806087 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.806526 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.806771 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:40:19.807857 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.788092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:39113 (local address 127.0.0.1:35154)
0809 22:40:19.788364 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.788385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.788505 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.788974 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.788979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.789396 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.789553 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.789561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.790718 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.790726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.793479 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.793507 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.795338 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.795344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.795495 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.796123 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.796143 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.799123 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.803831 (+ 4708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.803838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.803842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.804496 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.805033 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.805039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.805043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.805231 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.806049 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.806055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.806232 (+ 177us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.806398 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.807531 (+ 1133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260809 22:40:19.809037 11844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:40:19.809934 11687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:19.811249 11687 log.cc:824] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:19.814740 11687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b: No bootstrap required, opened a new log
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:40:19.825150 11687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:19.825754 11687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:19.825945 11687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 583fb8386b7240869326fadc3759f08b, State: Initialized, Role: FOLLOWER
I20260809 22:40:19.826733 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.810025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42270)
0809 22:40:19.811049 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.811066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.811200 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.811526 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.811531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.812084 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.812300 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.812312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.813532 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.813536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.814943 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.814955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.815989 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.815995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.816218 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.816855 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.816873 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.819562 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.823725 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.823732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.823737 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.824264 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.824751 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.824756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.824760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.824908 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.825557 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.825561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.825710 (+ 149us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.825917 (+ 207us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.826270 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931}
I20260809 22:40:19.827076 11687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:19.826951 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.810321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.28.243.124:34457)
0809 22:40:19.810563 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.810570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.811138 (+ 568us) server_negotiation.cc:407] Connection header received
0809 22:40:19.811324 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.811329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.811398 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.811508 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.812471 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.813402 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.815084 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.815802 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.815973 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.819778 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.819783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.819787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.819816 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.823551 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.824411 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.824417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.824420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.824619 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.825021 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.825029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.825033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.825399 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.825565 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.826112 (+ 547us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.826452 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":75}
I20260809 22:40:19.828044 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.808524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.28.243.125:39113)
0809 22:40:19.808780 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.808786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.808805 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:40:19.809119 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.809125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.809201 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.809298 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.810995 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.812110 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.813809 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.814535 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.814674 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.820247 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.820255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.820261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.820311 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.824389 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.825380 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.825386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.825390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.825586 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.826123 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.826129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.826132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.826510 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.826827 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.827438 (+ 611us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.827701 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260809 22:40:19.829246 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:19.829698 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:40:19.830991 11687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:19.834560 11844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: Bootstrap starting.
I20260809 22:40:19.835453 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:40:19.840122 11844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:19.842176 11844 log.cc:824] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:19.846942 11844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: No bootstrap required, opened a new log
W20260809 22:40:19.858451 11639 tablet.cc:2369] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:19.864705 11844 raft_consensus.cc:348] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:19.865608 11844 raft_consensus.cc:374] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:19.865839 11844 raft_consensus.cc:729] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f12c32b90c3b4e10a99bb9a2922ac8a6, State: Initialized, Role: FOLLOWER
I20260809 22:40:19.867095 11844 consensus_queue.cc:260] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
W20260809 22:40:19.874102 11872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:40:19.874357 11872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:40:19.874547 11876 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:19.874985 11876 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:40:19.876684 11844 sys_catalog.cc:564] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:19.857071 11616 master_runner.cc:186] Error getting master registration for 127.28.243.125:39113: OK, instance_id { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" instance_seqno: 1786315219178692 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 22:40:19.894166 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.851807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.28.243.124:34457)
0809 22:40:19.852143 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.852153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.862905 (+ 10752us) server_negotiation.cc:407] Connection header received
0809 22:40:19.863347 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.863355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.863453 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.863539 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.872613 (+ 9074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.873952 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.877560 (+ 3608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.878387 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.878501 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.885321 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.885330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.885336 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.885392 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.889988 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.891406 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.891414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.891418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.891596 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.892160 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.892167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.892171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.892607 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.892898 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.893484 (+ 586us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.893761 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 22:40:19.894980 11616 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 22:40:19.897291 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.847310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.28.243.126:39017)
0809 22:40:19.847576 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.847587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.854232 (+ 6645us) server_negotiation.cc:407] Connection header received
0809 22:40:19.868922 (+ 14690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.868931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.869040 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.869131 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.873262 (+ 4131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.874415 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.876895 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.877778 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.877922 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.884510 (+ 6588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.884529 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.884539 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.884657 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.890769 (+ 6112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.892153 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.892161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.892167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.892373 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.892989 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.892998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.893004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.893489 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.893823 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.894671 (+ 848us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.895075 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 22:40:19.908308 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.862138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.28.243.125:39113)
0809 22:40:19.862366 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.862373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.868647 (+ 6274us) server_negotiation.cc:407] Connection header received
0809 22:40:19.869150 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.869156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.869245 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.869327 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.877581 (+ 8254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.878792 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.892840 (+ 14048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.893558 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.893706 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.898618 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.898627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.898632 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.898682 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.903690 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.904865 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.904872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.904877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.905072 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.905590 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.905596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.905600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.906057 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.906354 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.907333 (+ 979us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.907704 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 22:40:19.918381 11765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } attempt: 5
W20260809 22:40:19.930919 11887 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:40:19.931172 11887 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 09 22:40:19 dist-test-slave-rgb1 krb5kdc[11602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315219, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
I20260809 22:40:19.933399 11853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:40:19.933753 11853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:19.935678 11853 leader_election.cc:288] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39b8748b026f4eed9eb2d3873fc5f564 (127.28.243.125:39113), f12c32b90c3b4e10a99bb9a2922ac8a6 (127.28.243.124:34457)
I20260809 22:40:19.937191 11818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583fb8386b7240869326fadc3759f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" is_pre_election: true
I20260809 22:40:19.937942 11818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 583fb8386b7240869326fadc3759f08b in term 0.
I20260809 22:40:19.938905 11635 leader_election.cc:302] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583fb8386b7240869326fadc3759f08b, f12c32b90c3b4e10a99bb9a2922ac8a6; no voters:
I20260809 22:40:19.940503 11853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:40:19.940721 11853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:40:19.940820 11853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:19.919194 11774 master_runner.cc:186] Error getting master registration for 127.28.243.125:39113: OK, instance_id { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" instance_seqno: 1786315219178692 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 22:40:19.943105 11774 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 22:40:19.944177 11853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:19.946049 11818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583fb8386b7240869326fadc3759f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6"
I20260809 22:40:19.946347 11818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:19.950301 11818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 583fb8386b7240869326fadc3759f08b in term 1.
I20260809 22:40:19.954303 11635 leader_election.cc:302] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 583fb8386b7240869326fadc3759f08b, f12c32b90c3b4e10a99bb9a2922ac8a6; no voters:
I20260809 22:40:19.955615 11853 leader_election.cc:288] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [CANDIDATE]: Term 1 election: Requested vote from peers 39b8748b026f4eed9eb2d3873fc5f564 (127.28.243.125:39113), f12c32b90c3b4e10a99bb9a2922ac8a6 (127.28.243.124:34457)
I20260809 22:40:19.955964 11889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:19.957674 11889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [term 1 LEADER]: Becoming Leader. State: Replica: 583fb8386b7240869326fadc3759f08b, State: Running, Role: LEADER
I20260809 22:40:19.959988 11889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:19.966765 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.919121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:34457 (local address 127.0.0.1:42290)
0809 22:40:19.919328 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:19.919352 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:19.919473 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:19.923535 (+ 4062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:19.923540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:19.924164 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:19.924394 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.924410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.926266 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.926274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.927885 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:19.927901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.929218 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.929226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:19.929476 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.931166 (+ 1690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:19.931195 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:19.934571 (+ 3376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:19.960699 (+ 26128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.960707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.960712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.961408 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.962050 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:19.962057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:19.962063 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:19.962278 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:19.963140 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:19.963146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:19.963343 (+ 197us) client_negotiation.cc:769] Sending connection context
0809 22:40:19.963545 (+ 202us) client_negotiation.cc:240] Negotiation successful
0809 22:40:19.963880 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 22:40:19.968349 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:19.922995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.28.243.124:34457)
0809 22:40:19.923194 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:19.923200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:19.923221 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:40:19.923280 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:19.923286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:19.923367 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:19.923506 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:19.924619 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.926053 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.928103 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:19.928990 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:19.934805 (+ 5815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:19.935065 (+ 260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:19.935073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:19.935080 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:19.935127 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:19.960415 (+ 25288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.961676 (+ 1261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.961682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.961686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.961881 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:19.962479 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:19.962486 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:19.962491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:19.962961 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:19.967233 (+ 4272us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:19.967400 (+ 167us) server_negotiation.cc:299] Negotiation successful
0809 22:40:19.967890 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 22:40:19.973502 11765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564: Bootstrap starting.
W20260809 22:40:19.975602 11796 tablet.cc:2369] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:19.979537 11765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:19.981596 11765 log.cc:824] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:19.986964 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 583fb8386b7240869326fadc3759f08b. Latest consensus state: current_term: 1 leader_uuid: "583fb8386b7240869326fadc3759f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:19.987541 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:19.991539 11765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564: No bootstrap required, opened a new log
I20260809 22:40:19.993151 11892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:19.995441 11892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:20.008482 11765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:20.016353 11765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:20.016609 11765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b8748b026f4eed9eb2d3873fc5f564, State: Initialized, Role: FOLLOWER
I20260809 22:40:20.017804 11765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } }
I20260809 22:40:20.023043 11818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [term 1 FOLLOWER]: Refusing update from remote peer 583fb8386b7240869326fadc3759f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:20.024462 11853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:20.025080 11894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:20.025527 11894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:40:20.027772 11765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:20.028461 11737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:20.032580 11738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583fb8386b7240869326fadc3759f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b8748b026f4eed9eb2d3873fc5f564"
I20260809 22:40:20.034186 11739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "583fb8386b7240869326fadc3759f08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39b8748b026f4eed9eb2d3873fc5f564" is_pre_election: true
I20260809 22:40:20.041260 11737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [term 1 FOLLOWER]: Refusing update from remote peer 583fb8386b7240869326fadc3759f08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:20.043853 11853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:20.054519 11876 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 583fb8386b7240869326fadc3759f08b. Latest consensus state: current_term: 1 leader_uuid: "583fb8386b7240869326fadc3759f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:20.054859 11876 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:40:20.067334 11876 sys_catalog.cc:455] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "583fb8386b7240869326fadc3759f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:20.067674 11876 sys_catalog.cc:458] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:40:20.073066 11894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 583fb8386b7240869326fadc3759f08b. Latest consensus state: current_term: 1 leader_uuid: "583fb8386b7240869326fadc3759f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:20.073515 11894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:40:20.078224 11889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "583fb8386b7240869326fadc3759f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:20.078629 11889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:20.080539 11889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "583fb8386b7240869326fadc3759f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:20.080880 11889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:20.086356 11892 catalog_manager.cc:1347] Generated new cluster ID: 65272288cb21496f85fead541cada107
I20260809 22:40:20.086501 11892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:40:20.111495 11894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "583fb8386b7240869326fadc3759f08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "583fb8386b7240869326fadc3759f08b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39017 } } peers { permanent_uuid: "39b8748b026f4eed9eb2d3873fc5f564" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 39113 } } peers { permanent_uuid: "f12c32b90c3b4e10a99bb9a2922ac8a6" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 34457 } } }
I20260809 22:40:20.111951 11894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:40:20.137565 11892 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:40:20.139859 11892 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:40:20.150203 11912 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:40:20.154546 11892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 583fb8386b7240869326fadc3759f08b: Generated new TSK 0
I20260809 22:40:20.156097 11892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:40:20.933348 11887 catalog_manager.cc:1259] Loaded cluster ID: 65272288cb21496f85fead541cada107
I20260809 22:40:20.933458 11887 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: loading cluster ID for follower catalog manager: success
I20260809 22:40:20.935844 11887 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: acquiring CA information for follower catalog manager: success
I20260809 22:40:20.937834 11887 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f12c32b90c3b4e10a99bb9a2922ac8a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:40:21.154837 11912 catalog_manager.cc:1259] Loaded cluster ID: 65272288cb21496f85fead541cada107
I20260809 22:40:21.155012 11912 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564: loading cluster ID for follower catalog manager: success
I20260809 22:40:21.157871 11912 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564: acquiring CA information for follower catalog manager: success
I20260809 22:40:21.159940 11912 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39b8748b026f4eed9eb2d3873fc5f564: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:40:21.255168 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 11616
I20260809 22:40:21.256508 11682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:21.379782 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 11616
I20260809 22:40:21.417421 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 11694
I20260809 22:40:21.419205 11760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:21.533618 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 11694
I20260809 22:40:21.551970 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 11774
I20260809 22:40:21.553225 11839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:21.666106 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 11774
2026-08-09T22:40:21Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5040 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:40:21 dist-test-slave-rgb1 krb5kdc[11925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:40:21 dist-test-slave-rgb1 krb5kdc[11925](info): set up 2 sockets
Aug 09 22:40:21 dist-test-slave-rgb1 krb5kdc[11925](info): commencing operation
krb5kdc: starting...
W20260809 22:40:22.914191 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:40:23 dist-test-slave-rgb1 krb5kdc[11925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:40:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:40:23.257887 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39173
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:23.645186 11940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:23.645630 11940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:23.645805 11940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:23.653612 11940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:23.653793 11940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:23.653913 11940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:23.654017 11940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:23.654109 11940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:23.664712 11940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:23.666206 11940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:23.667747 11940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:23.682350 11945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:23.682392 11946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:23.682312 11948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:23.684672 11940 server_base.cc:1034] running on GCE node
I20260809 22:40:23.685670 11940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:23.687345 11940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:23.688755 11940 hybrid_clock.cc:648] HybridClock initialized: now 1786315223688692 us; error 42 us; skew 500 ppm
I20260809 22:40:23.689240 11940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:23 dist-test-slave-rgb1 krb5kdc[11925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315223, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:23.693094 11940 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:23.694939 11940 webserver.cc:460] Webserver started at http://127.28.243.126:35743/ using document root <none> and password file <none>
I20260809 22:40:23.695672 11940 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:23.695785 11940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:23.696064 11940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:23.699361 11940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "7c347172f3d247349dc5e91523689967"
format_stamp: "Formatted at 2026-08-09 22:40:23 on dist-test-slave-rgb1"
server_key: "14b34414a95c65d0686a2d2ca4c9de74"
server_key_iv: "b0cf9019641fabae8a9729291aafd85e"
server_key_version: "encryptionkey@0"
I20260809 22:40:23.700307 11940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "7c347172f3d247349dc5e91523689967"
format_stamp: "Formatted at 2026-08-09 22:40:23 on dist-test-slave-rgb1"
server_key: "14b34414a95c65d0686a2d2ca4c9de74"
server_key_iv: "b0cf9019641fabae8a9729291aafd85e"
server_key_version: "encryptionkey@0"
I20260809 22:40:23.706097 11940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 22:40:23.709939 11958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:23.711805 11940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260809 22:40:23.712018 11940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "7c347172f3d247349dc5e91523689967"
format_stamp: "Formatted at 2026-08-09 22:40:23 on dist-test-slave-rgb1"
server_key: "14b34414a95c65d0686a2d2ca4c9de74"
server_key_iv: "b0cf9019641fabae8a9729291aafd85e"
server_key_version: "encryptionkey@0"
I20260809 22:40:23.712221 11940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:23.729396 11940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:23.734686 11940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:23.735023 11940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:23.757119 11940 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:39173
I20260809 22:40:23.757165 12015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:39173 every 8 connection(s)
I20260809 22:40:23.758666 11940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:23.762761 12016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:23.764945 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 11940
I20260809 22:40:23.765161 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:40:23.765870 29645 external_mini_cluster.cc:1428] Setting key 3e996e3e83764ffa424007068ee3f45e
I20260809 22:40:23.784122 12016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967: Bootstrap starting.
Aug 09 22:40:23 dist-test-slave-rgb1 krb5kdc[11925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:23.787869 12016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:23.789336 12016 log.cc:824] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:23.793205 12016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967: No bootstrap required, opened a new log
I20260809 22:40:23.798158 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:23.768453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.28.243.126:39173)
0809 22:40:23.769590 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:23.769609 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:23.769670 (+ 61us) server_negotiation.cc:407] Connection header received
0809 22:40:23.772301 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:23.772337 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:23.773018 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:23.773564 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:23.776115 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:23.778939 (+ 2824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:23.781375 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:23.782998 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:23.783254 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:23.787948 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:23.787957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:23.787976 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:23.788060 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:23.792213 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:23.793194 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:23.793205 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:23.793213 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:23.793460 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:23.794265 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:23.794270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:23.794274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:23.795104 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:23.795453 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:23.795952 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 22:40:23.796507 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":378,"threads_started":1}
I20260809 22:40:23.802920 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:40:23.804242 12016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c347172f3d247349dc5e91523689967" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39173 } }
I20260809 22:40:23.804831 12016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:23.804947 12016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c347172f3d247349dc5e91523689967, State: Initialized, Role: FOLLOWER
I20260809 22:40:23.805684 12016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c347172f3d247349dc5e91523689967" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39173 } }
I20260809 22:40:23.806034 12016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:40:23.806205 12016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:40:23.806452 12016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:23.809525 12016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c347172f3d247349dc5e91523689967" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39173 } }
I20260809 22:40:23.810115 12016 leader_election.cc:302] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c347172f3d247349dc5e91523689967; no voters:
I20260809 22:40:23.810997 12016 leader_election.cc:288] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:40:23.811240 12022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:23.812285 12022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [term 1 LEADER]: Becoming Leader. State: Replica: 7c347172f3d247349dc5e91523689967, State: Running, Role: LEADER
I20260809 22:40:23.812901 12022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c347172f3d247349dc5e91523689967" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39173 } }
I20260809 22:40:23.813849 12016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:23.817342 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c347172f3d247349dc5e91523689967. Latest consensus state: current_term: 1 leader_uuid: "7c347172f3d247349dc5e91523689967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c347172f3d247349dc5e91523689967" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39173 } } }
I20260809 22:40:23.817315 12023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c347172f3d247349dc5e91523689967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c347172f3d247349dc5e91523689967" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39173 } } }
I20260809 22:40:23.817891 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:23.817976 12023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967 [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:23.821197 12031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:23.825964 12031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:23.838104 12031 catalog_manager.cc:1347] Generated new cluster ID: 56b7e7752fb3455abe8c87e752108ec8
I20260809 22:40:23.838271 12031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:40:23.882156 12031 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:40:23.884085 12031 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:40:23.901089 12031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c347172f3d247349dc5e91523689967: Generated new TSK 0
I20260809 22:40:23.902348 12031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:40:23.919920 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 11940
I20260809 22:40:23.921443 12010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:24.030084 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 11940
2026-08-09T22:40:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2357 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:40:24 dist-test-slave-rgb1 krb5kdc[12049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:40:24 dist-test-slave-rgb1 krb5kdc[12049](info): set up 2 sockets
Aug 09 22:40:24 dist-test-slave-rgb1 krb5kdc[12049](info): commencing operation
krb5kdc: starting...
W20260809 22:40:25.371176 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:40:25 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:40:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:40:25.727577 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33071
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:26.112392 12062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:26.112810 12062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:26.113000 12062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:26.121695 12062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:26.121848 12062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:26.121922 12062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:26.122021 12062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:26.122097 12062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:26.133901 12062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:26.135507 12062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:26.137082 12062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:26.150810 12067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:26.150812 12071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:26.151134 12068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:26.152695 12062 server_base.cc:1034] running on GCE node
I20260809 22:40:26.153739 12062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:26.155476 12062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:26.156724 12062 hybrid_clock.cc:648] HybridClock initialized: now 1786315226156718 us; error 109 us; skew 500 ppm
I20260809 22:40:26.157189 12062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:26 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:26.161731 12062 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:26.163770 12062 webserver.cc:460] Webserver started at http://127.28.243.126:42237/ using document root <none> and password file <none>
I20260809 22:40:26.164636 12062 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:26.164806 12062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:26.165158 12062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:26.168455 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "219ca4c356e0408aa821cd288b4ae82d"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "26ddbf990604feaca7d4c1ebb7ea2735"
server_key_iv: "786853ed417a086e072168edd53e71f7"
server_key_version: "encryptionkey@0"
I20260809 22:40:26.169385 12062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "219ca4c356e0408aa821cd288b4ae82d"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "26ddbf990604feaca7d4c1ebb7ea2735"
server_key_iv: "786853ed417a086e072168edd53e71f7"
server_key_version: "encryptionkey@0"
I20260809 22:40:26.178153 12062 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260809 22:40:26.182538 12078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:26.184938 12062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:40:26.185187 12062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "219ca4c356e0408aa821cd288b4ae82d"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "26ddbf990604feaca7d4c1ebb7ea2735"
server_key_iv: "786853ed417a086e072168edd53e71f7"
server_key_version: "encryptionkey@0"
I20260809 22:40:26.185452 12062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:26.207698 12062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:26.213835 12062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:26.214180 12062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:26.240140 12062 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33071
I20260809 22:40:26.240175 12136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33071 every 8 connection(s)
I20260809 22:40:26.241828 12062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:26.246477 12137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:26.248704 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 12062
I20260809 22:40:26.248982 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:40:26.249912 29645 external_mini_cluster.cc:1428] Setting key 0cf795b32c2ed4868dfeebc19dc00d1f
I20260809 22:40:26.263201 12137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Bootstrap starting.
I20260809 22:40:26.270092 12137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Neither blocks nor log segments found. Creating new log.
Aug 09 22:40:26 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:26.271553 12137 log.cc:824] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:26.277895 12137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: No bootstrap required, opened a new log
I20260809 22:40:26.287298 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:26.252786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.28.243.126:33071)
0809 22:40:26.253790 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:26.253804 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:26.253865 (+ 61us) server_negotiation.cc:407] Connection header received
0809 22:40:26.256854 (+ 2989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:26.256886 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:26.257410 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:26.257825 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:26.260345 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:26.261992 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:26.264432 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:26.265375 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:26.265533 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:26.273314 (+ 7781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:26.273327 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:26.273352 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:26.273417 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:26.281308 (+ 7891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:26.282985 (+ 1677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:26.282994 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:26.283002 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:26.283239 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:26.283708 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:26.283713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:26.283717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:26.284549 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:26.284962 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:26.285565 (+ 603us) server_negotiation.cc:299] Negotiation successful
0809 22:40:26.286092 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":485,"threads_started":1}
I20260809 22:40:26.294080 12137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:26.294641 12137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:26.294763 12137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219ca4c356e0408aa821cd288b4ae82d, State: Initialized, Role: FOLLOWER
I20260809 22:40:26.295850 12137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:26.296200 12137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:40:26.296367 12137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:40:26.296880 12137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:26.300313 12137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:26.301110 12137 leader_election.cc:302] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219ca4c356e0408aa821cd288b4ae82d; no voters:
I20260809 22:40:26.302055 12137 leader_election.cc:288] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:40:26.302357 12142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:26.303705 12142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 LEADER]: Becoming Leader. State: Replica: 219ca4c356e0408aa821cd288b4ae82d, State: Running, Role: LEADER
I20260809 22:40:26.304278 12142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:26.304730 12137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:26.308759 12143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "219ca4c356e0408aa821cd288b4ae82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } } }
I20260809 22:40:26.309257 12143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:26.310153 12144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219ca4c356e0408aa821cd288b4ae82d. Latest consensus state: current_term: 1 leader_uuid: "219ca4c356e0408aa821cd288b4ae82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } } }
I20260809 22:40:26.310526 12144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:26.320472 12149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:26.327178 12149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:26.338019 12149 catalog_manager.cc:1347] Generated new cluster ID: 007c20da25844fba80a29406d8bc8d87
I20260809 22:40:26.338165 12149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:40:26.373095 12149 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:40:26.374562 12149 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:40:26.384193 12149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Generated new TSK 0
I20260809 22:40:26.385205 12149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:40:26.490898 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33071
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:26.881259 12166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:26.881579 12166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:26.881685 12166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:26.890204 12166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:26.890338 12166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:26.890473 12166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:40:26.902287 12166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:26.903695 12166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:26.905536 12166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:26.921658 12172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:26.921669 12173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:26.922574 12175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:26.923226 12166 server_base.cc:1034] running on GCE node
I20260809 22:40:26.923985 12166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:26.925269 12166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:26.926498 12166 hybrid_clock.cc:648] HybridClock initialized: now 1786315226926413 us; error 78 us; skew 500 ppm
I20260809 22:40:26.926875 12166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:26 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:26.935015 12166 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:40:26.937248 12166 webserver.cc:460] Webserver started at http://127.28.243.65:33899/ using document root <none> and password file <none>
I20260809 22:40:26.938045 12166 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:26.938217 12166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:26.938558 12166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:26.941928 12166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "2b616e5c3bea481db474894bbdae10c5"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "dfd588e8c3a56212a35dbf541e62b4cf"
server_key_iv: "1cb878710285492a630bfe4a700841f1"
server_key_version: "encryptionkey@0"
I20260809 22:40:26.942833 12166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "2b616e5c3bea481db474894bbdae10c5"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "dfd588e8c3a56212a35dbf541e62b4cf"
server_key_iv: "1cb878710285492a630bfe4a700841f1"
server_key_version: "encryptionkey@0"
I20260809 22:40:26.949229 12166 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 22:40:26.953732 12182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:26.955448 12166 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:40:26.955703 12166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "2b616e5c3bea481db474894bbdae10c5"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "dfd588e8c3a56212a35dbf541e62b4cf"
server_key_iv: "1cb878710285492a630bfe4a700841f1"
server_key_version: "encryptionkey@0"
I20260809 22:40:26.956007 12166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:26.977610 12166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:26.983532 12166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:26.983955 12166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:26.985183 12166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:26.986791 12166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:26.986922 12166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:26.987093 12166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:26.987179 12166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:27.028384 12166 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:37833
I20260809 22:40:27.028487 12307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:37833 every 8 connection(s)
I20260809 22:40:27.029839 12166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:40:27.033540 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 12166
I20260809 22:40:27.033847 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:40:27.034719 29645 external_mini_cluster.cc:1428] Setting key f5ffa2c2e98f48388977957e34489ee5
Aug 09 22:40:27 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315226, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:27.053395 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:27.032743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:43905 (local address 127.28.243.126:33071)
0809 22:40:27.033296 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:27.033301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:27.034628 (+ 1327us) server_negotiation.cc:407] Connection header received
0809 22:40:27.036891 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:27.036896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:27.036987 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:27.037299 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:27.039127 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.039903 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.041638 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.042051 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.042204 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:27.046370 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:27.046377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:27.046381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:27.046421 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:27.049625 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:27.050541 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:27.050552 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:27.050556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:27.050697 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:27.051220 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:27.051236 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:27.051239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:27.051568 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:27.051783 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:27.052649 (+ 866us) server_negotiation.cc:299] Negotiation successful
0809 22:40:27.052957 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":340,"threads_started":1}
I20260809 22:40:27.054633 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:27.033249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:43905)
0809 22:40:27.034446 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:27.034499 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:27.036542 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:27.037606 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:27.037615 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:27.038257 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:27.038927 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:27.038946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.040092 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.040096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:27.041452 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:27.041465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.042323 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.042330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:27.042583 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:27.043353 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:27.043384 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:27.046069 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:27.049817 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:27.049828 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:27.049849 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:27.050363 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:27.050836 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:27.050842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:27.050846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:27.051075 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:27.051725 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:27.051734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:27.052160 (+ 426us) client_negotiation.cc:769] Sending connection context
0809 22:40:27.052495 (+ 335us) client_negotiation.cc:240] Negotiation successful
0809 22:40:27.053169 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":412,"threads_started":1}
I20260809 22:40:27.056856 12309 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33071
I20260809 22:40:27.057251 12309 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:27.058399 12309 heartbeater.cc:511] Master 127.28.243.126:33071 requested a full tablet report, sending...
I20260809 22:40:27.061239 12095 ts_manager.cc:194] Registered new tserver with Master: 2b616e5c3bea481db474894bbdae10c5 (127.28.243.65:37833)
I20260809 22:40:27.062747 12095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:43905
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:40:27.124961 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33071
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:27.503877 12315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:27.504262 12315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:27.504439 12315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:27.512310 12315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:27.512493 12315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:27.512701 12315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:40:27.524041 12315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:27.525621 12315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:27.527136 12315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:27.542680 12321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:27.542754 12323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:27.542707 12320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:27.543483 12315 server_base.cc:1034] running on GCE node
I20260809 22:40:27.544152 12315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:27.545271 12315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:27.546505 12315 hybrid_clock.cc:648] HybridClock initialized: now 1786315227546469 us; error 35 us; skew 500 ppm
I20260809 22:40:27.546876 12315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:27 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:27.550710 12315 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:40:27.552500 12315 webserver.cc:460] Webserver started at http://127.28.243.66:33093/ using document root <none> and password file <none>
I20260809 22:40:27.553290 12315 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:27.553462 12315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:27.553838 12315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:27.557205 12315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "74c8140a399d4e25bbb671a3b5ad2530"
format_stamp: "Formatted at 2026-08-09 22:40:27 on dist-test-slave-rgb1"
server_key: "f881d00bfec72027998cc65168673a8f"
server_key_iv: "74a70782a0d5785445275d7b4e8150e3"
server_key_version: "encryptionkey@0"
I20260809 22:40:27.558068 12315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "74c8140a399d4e25bbb671a3b5ad2530"
format_stamp: "Formatted at 2026-08-09 22:40:27 on dist-test-slave-rgb1"
server_key: "f881d00bfec72027998cc65168673a8f"
server_key_iv: "74a70782a0d5785445275d7b4e8150e3"
server_key_version: "encryptionkey@0"
I20260809 22:40:27.563939 12315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 22:40:27.568169 12331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:27.569952 12315 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:40:27.570163 12315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "74c8140a399d4e25bbb671a3b5ad2530"
format_stamp: "Formatted at 2026-08-09 22:40:27 on dist-test-slave-rgb1"
server_key: "f881d00bfec72027998cc65168673a8f"
server_key_iv: "74a70782a0d5785445275d7b4e8150e3"
server_key_version: "encryptionkey@0"
I20260809 22:40:27.570406 12315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:27.599432 12315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:27.604777 12315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:27.605216 12315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:27.606546 12315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:27.608125 12315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:27.608234 12315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:27.608392 12315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:27.608466 12315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:27.648423 12315 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39289
I20260809 22:40:27.648459 12453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39289 every 8 connection(s)
I20260809 22:40:27.649962 12315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:40:27.659722 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 12315
I20260809 22:40:27.660075 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:40:27.660633 29645 external_mini_cluster.cc:1428] Setting key d2abfa21d4ed0a0db3a6ec7b424d10a5
Aug 09 22:40:27 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315227, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:27.674274 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:27.652977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:49695 (local address 127.28.243.126:33071)
0809 22:40:27.653490 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:27.653496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:27.654465 (+ 969us) server_negotiation.cc:407] Connection header received
0809 22:40:27.656656 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:27.656663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:27.656760 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:27.656901 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:27.658595 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.659411 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.661881 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.662616 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.662772 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:27.667493 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:27.667499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:27.667502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:27.667540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:27.670577 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:27.671648 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:27.671654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:27.671658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:27.671805 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:27.672270 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:27.672280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:27.672282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:27.672680 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:27.672858 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:27.673628 (+ 770us) server_negotiation.cc:299] Negotiation successful
0809 22:40:27.673877 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":386,"thread_start_us":257,"threads_started":1}
I20260809 22:40:27.675467 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:27.653424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:49695)
0809 22:40:27.654282 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:27.654325 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:27.656280 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:27.657172 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:27.657181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:27.657758 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:27.658399 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:27.658419 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.659601 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.659606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:27.661694 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:27.661709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:27.662898 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:27.662909 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:27.663210 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:27.664114 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:27.664140 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:27.667176 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:27.670975 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:27.670985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:27.671000 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:27.671499 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:27.671933 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:27.671938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:27.671942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:27.672139 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:27.672821 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:27.672832 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:27.673218 (+ 386us) client_negotiation.cc:769] Sending connection context
0809 22:40:27.673505 (+ 287us) client_negotiation.cc:240] Negotiation successful
0809 22:40:27.674104 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":379,"threads_started":1}
I20260809 22:40:27.677112 12454 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33071
I20260809 22:40:27.677482 12454 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:27.678442 12454 heartbeater.cc:511] Master 127.28.243.126:33071 requested a full tablet report, sending...
I20260809 22:40:27.680281 12095 ts_manager.cc:194] Registered new tserver with Master: 74c8140a399d4e25bbb671a3b5ad2530 (127.28.243.66:39289)
I20260809 22:40:27.681231 12095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:49695
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:40:27.734989 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33071
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:28.066869 12309 heartbeater.cc:503] Master 127.28.243.126:33071 was elected leader, sending a full tablet report...
W20260809 22:40:28.123503 12460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:28.123831 12460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:28.124014 12460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:28.132442 12460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:28.132721 12460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:28.132959 12460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:40:28.144816 12460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:28.146337 12460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:28.147846 12460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:28.164008 12466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:28.163987 12465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:28.164136 12468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:28.164968 12460 server_base.cc:1034] running on GCE node
I20260809 22:40:28.165678 12460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:28.166771 12460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:28.168045 12460 hybrid_clock.cc:648] HybridClock initialized: now 1786315228167972 us; error 80 us; skew 500 ppm
I20260809 22:40:28.168418 12460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:28 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:28.172714 12460 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:40:28.174566 12460 webserver.cc:460] Webserver started at http://127.28.243.67:43317/ using document root <none> and password file <none>
I20260809 22:40:28.175501 12460 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:28.175659 12460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:28.176095 12460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:28.179757 12460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "c03a088f961b4a108c60321f4f806408"
format_stamp: "Formatted at 2026-08-09 22:40:28 on dist-test-slave-rgb1"
server_key: "57f4fb94cee384f7520b19f3b3cc8934"
server_key_iv: "8cfb59c17c18b47438eefa1d6a1add1d"
server_key_version: "encryptionkey@0"
I20260809 22:40:28.180724 12460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "c03a088f961b4a108c60321f4f806408"
format_stamp: "Formatted at 2026-08-09 22:40:28 on dist-test-slave-rgb1"
server_key: "57f4fb94cee384f7520b19f3b3cc8934"
server_key_iv: "8cfb59c17c18b47438eefa1d6a1add1d"
server_key_version: "encryptionkey@0"
I20260809 22:40:28.186638 12460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 22:40:28.191272 12475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:28.193219 12460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 22:40:28.193449 12460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "c03a088f961b4a108c60321f4f806408"
format_stamp: "Formatted at 2026-08-09 22:40:28 on dist-test-slave-rgb1"
server_key: "57f4fb94cee384f7520b19f3b3cc8934"
server_key_iv: "8cfb59c17c18b47438eefa1d6a1add1d"
server_key_version: "encryptionkey@0"
I20260809 22:40:28.193717 12460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:28.213893 12460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:28.218912 12460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:28.219529 12460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:28.221174 12460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:28.222949 12460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:28.223045 12460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:28.223142 12460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:28.223188 12460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:28.262372 12460 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:40905
I20260809 22:40:28.262395 12592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:40905 every 8 connection(s)
I20260809 22:40:28.263728 12460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:40:28.269721 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 12460
I20260809 22:40:28.270170 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:40:28.270913 29645 external_mini_cluster.cc:1428] Setting key 7dded1bee4c9aedd782133d999e6a31e
Aug 09 22:40:28 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315228, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:28.291446 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.266619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:52317 (local address 127.28.243.126:33071)
0809 22:40:28.267219 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.267227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.269150 (+ 1923us) server_negotiation.cc:407] Connection header received
0809 22:40:28.272530 (+ 3380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.272537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.272670 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.272826 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:28.275717 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.277178 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.279170 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.279617 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.279731 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.283604 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:28.283610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:28.283614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:28.283664 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:28.286810 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.288035 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.288039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.288042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.288291 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.288997 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.289002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.289006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.289485 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:28.289622 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.290785 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.291145 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":270,"threads_started":1}
I20260809 22:40:28.292498 12596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.267364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:52317)
0809 22:40:28.268915 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:28.268970 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:28.272110 (+ 3140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:28.273238 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:28.273253 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:28.274088 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:28.275468 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.275512 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.277413 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.277417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.278984 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.279000 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.279876 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.279882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.280123 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.281003 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:28.281033 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:28.283341 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:28.287069 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:28.287080 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:28.287100 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:28.287883 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:28.288465 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:28.288471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:28.288476 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:28.288775 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:28.289679 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:28.289687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:28.290311 (+ 624us) client_negotiation.cc:769] Sending connection context
0809 22:40:28.290647 (+ 336us) client_negotiation.cc:240] Negotiation successful
0809 22:40:28.291145 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":963,"threads_started":1}
I20260809 22:40:28.294587 12593 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33071
I20260809 22:40:28.294957 12593 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:28.295787 12593 heartbeater.cc:511] Master 127.28.243.126:33071 requested a full tablet report, sending...
I20260809 22:40:28.297575 12095 ts_manager.cc:194] Registered new tserver with Master: c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905)
I20260809 22:40:28.298516 12095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:52317
I20260809 22:40:28.302266 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:40:28.322871 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.307905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.28.243.126:33071)
0809 22:40:28.308156 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.308167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.308642 (+ 475us) server_negotiation.cc:407] Connection header received
0809 22:40:28.308838 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.308842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.308930 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.309046 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:28.310589 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.311426 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.313023 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.313545 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.313697 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.316410 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:28.316416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:28.316420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:28.316456 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:28.319603 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.320419 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.320423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.320426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.320671 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.321161 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.321166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.321168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.321568 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:28.321806 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.322248 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.322440 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:40:28.328277 12095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:40:28.335494 12095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:40:28.363958 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.351128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.28.243.67:40905)
0809 22:40:28.353187 (+ 2059us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.353196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.353972 (+ 776us) server_negotiation.cc:407] Connection header received
0809 22:40:28.354092 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.354102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.354361 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.354577 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:28.355637 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.357877 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.360864 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.362252 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.362412 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.362674 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.363338 (+ 664us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.363540 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1869,"thread_start_us":1674,"threads_started":1}
I20260809 22:40:28.364545 12606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.350076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39289 (local address 127.0.0.1:38498)
0809 22:40:28.352201 (+ 2125us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:28.352254 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:28.352450 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:28.353798 (+ 1348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:28.353807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:28.353834 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:28.354392 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.354411 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.357017 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.357024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.360553 (+ 3529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.360598 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.362961 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.362969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.363152 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.363190 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 22:40:28.363286 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 22:40:28.363519 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1847,"mutex_wait_us":45,"thread_start_us":1882,"threads_started":1}
I20260809 22:40:28.364594 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.350771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:40905 (local address 127.0.0.1:46912)
0809 22:40:28.353659 (+ 2888us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:28.353681 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:28.353815 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:28.354893 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:28.354898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:28.354928 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:28.355397 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.355415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.358072 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.358080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.360715 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.360730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.362813 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.362819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.363066 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.363109 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 22:40:28.363273 (+ 164us) client_negotiation.cc:240] Negotiation successful
0809 22:40:28.363540 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2742,"mutex_wait_us":63,"thread_start_us":2804,"threads_started":1}
I20260809 22:40:28.364841 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.350325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.28.243.66:39289)
0809 22:40:28.351709 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.351722 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.352642 (+ 920us) server_negotiation.cc:407] Connection header received
0809 22:40:28.352892 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.352902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.353188 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.353421 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:28.354659 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.356820 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.360781 (+ 3961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.362457 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.362610 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.362796 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.364207 (+ 1411us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.364456 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":1048,"threads_started":1}
I20260809 22:40:28.365718 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.352489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59946 (local address 127.28.243.65:37833)
0809 22:40:28.355309 (+ 2820us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.355321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.355981 (+ 660us) server_negotiation.cc:407] Connection header received
0809 22:40:28.356217 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.356226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.356475 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.356723 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:28.358192 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.359413 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.362165 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.363769 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.363971 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.364331 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.365193 (+ 862us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.365384 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2586,"thread_start_us":2364,"threads_started":1}
I20260809 22:40:28.365368 12610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.351937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37833 (local address 127.0.0.1:59946)
0809 22:40:28.355852 (+ 3915us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:28.355869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:28.356033 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:28.357015 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:28.357019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:28.357043 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:28.357951 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.357964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.359567 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.359573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.361938 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.361954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.364624 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.364631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.364810 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.364854 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 22:40:28.364962 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 22:40:28.365104 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3758,"mutex_wait_us":106,"thread_start_us":3602,"threads_started":1}
I20260809 22:40:28.369800 12235 tablet_service.cc:1467] Processing CreateTablet for tablet a46c4533e83d47ee994dbaf7e02ab3f4 (DEFAULT_TABLE table=test-table [id=75096e45c37e40cba253ec84453617dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:28.369850 12381 tablet_service.cc:1467] Processing CreateTablet for tablet a46c4533e83d47ee994dbaf7e02ab3f4 (DEFAULT_TABLE table=test-table [id=75096e45c37e40cba253ec84453617dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:28.370397 12524 tablet_service.cc:1467] Processing CreateTablet for tablet a46c4533e83d47ee994dbaf7e02ab3f4 (DEFAULT_TABLE table=test-table [id=75096e45c37e40cba253ec84453617dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:28.371428 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46c4533e83d47ee994dbaf7e02ab3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:28.371508 12381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46c4533e83d47ee994dbaf7e02ab3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:28.371824 12524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a46c4533e83d47ee994dbaf7e02ab3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:28.389986 12613 tablet_bootstrap.cc:492] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Bootstrap starting.
I20260809 22:40:28.393369 12612 tablet_bootstrap.cc:492] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Bootstrap starting.
I20260809 22:40:28.397249 12613 tablet_bootstrap.cc:654] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:28.397380 12612 tablet_bootstrap.cc:654] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:28.398516 12614 tablet_bootstrap.cc:492] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Bootstrap starting.
I20260809 22:40:28.398849 12612 log.cc:824] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:28.398849 12613 log.cc:824] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:28.402216 12613 tablet_bootstrap.cc:492] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: No bootstrap required, opened a new log
I20260809 22:40:28.402549 12612 tablet_bootstrap.cc:492] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: No bootstrap required, opened a new log
I20260809 22:40:28.402829 12612 ts_tablet_manager.cc:1397] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260809 22:40:28.402853 12613 ts_tablet_manager.cc:1397] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260809 22:40:28.403301 12614 tablet_bootstrap.cc:654] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:28.404794 12614 log.cc:824] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:28.408443 12614 tablet_bootstrap.cc:492] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: No bootstrap required, opened a new log
I20260809 22:40:28.408803 12614 ts_tablet_manager.cc:1397] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260809 22:40:28.412061 12613 raft_consensus.cc:348] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.412532 12613 raft_consensus.cc:374] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:28.412688 12613 raft_consensus.cc:729] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03a088f961b4a108c60321f4f806408, State: Initialized, Role: FOLLOWER
I20260809 22:40:28.413429 12613 consensus_queue.cc:260] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.413653 12612 raft_consensus.cc:348] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.414103 12612 raft_consensus.cc:374] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:28.414211 12612 raft_consensus.cc:729] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b616e5c3bea481db474894bbdae10c5, State: Initialized, Role: FOLLOWER
I20260809 22:40:28.415014 12593 heartbeater.cc:503] Master 127.28.243.126:33071 was elected leader, sending a full tablet report...
I20260809 22:40:28.414954 12612 consensus_queue.cc:260] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.415797 12613 ts_tablet_manager.cc:1428] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260809 22:40:28.418877 12612 ts_tablet_manager.cc:1428] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260809 22:40:28.421187 12614 raft_consensus.cc:348] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.421761 12614 raft_consensus.cc:374] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:28.421878 12614 raft_consensus.cc:729] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8140a399d4e25bbb671a3b5ad2530, State: Initialized, Role: FOLLOWER
I20260809 22:40:28.422639 12614 consensus_queue.cc:260] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.424081 12454 heartbeater.cc:503] Master 127.28.243.126:33071 was elected leader, sending a full tablet report...
I20260809 22:40:28.425086 12614 ts_tablet_manager.cc:1428] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260809 22:40:28.515669 12594 tablet.cc:2369] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:28.525955 12619 raft_consensus.cc:482] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:40:28.526321 12619 raft_consensus.cc:504] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.528496 12619 leader_election.cc:288] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c8140a399d4e25bbb671a3b5ad2530 (127.28.243.66:39289), c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905)
W20260809 22:40:28.532769 12310 tablet.cc:2369] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:28.537828 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.528894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:45851 (local address 127.28.243.66:39289)
0809 22:40:28.529310 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.529319 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.529974 (+ 655us) server_negotiation.cc:407] Connection header received
0809 22:40:28.530371 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.530378 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.530461 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.530580 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:28.531423 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.532500 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.534881 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.536134 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.536291 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.536368 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.537279 (+ 911us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.537562 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260809 22:40:28.538241 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.528884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39289 (local address 127.28.243.65:45851)
0809 22:40:28.529830 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:28.529857 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:28.530192 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:28.530596 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:28.530601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:28.530646 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:28.531225 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.531238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.532750 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.532755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.534670 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.534685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.536520 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.536529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.536975 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.537030 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 22:40:28.537314 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 22:40:28.537540 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":97,"spinlock_wait_cycles":21376,"thread_start_us":411,"threads_started":1}
I20260809 22:40:28.539052 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.529669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:54971 (local address 127.28.243.67:40905)
0809 22:40:28.529944 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.529953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.530302 (+ 349us) server_negotiation.cc:407] Connection header received
0809 22:40:28.530550 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.530562 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.530633 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.530762 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:28.531733 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.532968 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.535856 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.537759 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.538017 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.538108 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.538625 (+ 517us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.538815 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260809 22:40:28.539055 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.528932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:40905 (local address 127.28.243.65:54971)
0809 22:40:28.530167 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:28.530185 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:28.530386 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:28.530810 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:28.530817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:28.530867 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:28.531477 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.531491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.533191 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.533197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.535487 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:28.535505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.538160 (+ 2655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.538170 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:28.538417 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.538461 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 22:40:28.538566 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 22:40:28.538702 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":28,"thread_start_us":847,"threads_started":1}
I20260809 22:40:28.539520 12403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46c4533e83d47ee994dbaf7e02ab3f4" candidate_uuid: "2b616e5c3bea481db474894bbdae10c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8140a399d4e25bbb671a3b5ad2530" is_pre_election: true
I20260809 22:40:28.540117 12403 raft_consensus.cc:2454] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b616e5c3bea481db474894bbdae10c5 in term 0.
I20260809 22:40:28.540270 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46c4533e83d47ee994dbaf7e02ab3f4" candidate_uuid: "2b616e5c3bea481db474894bbdae10c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03a088f961b4a108c60321f4f806408" is_pre_election: true
I20260809 22:40:28.540858 12546 raft_consensus.cc:2454] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b616e5c3bea481db474894bbdae10c5 in term 0.
I20260809 22:40:28.541136 12187 leader_election.cc:302] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b616e5c3bea481db474894bbdae10c5, 74c8140a399d4e25bbb671a3b5ad2530; no voters:
I20260809 22:40:28.541702 12619 raft_consensus.cc:2781] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:40:28.541888 12619 raft_consensus.cc:482] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:40:28.542096 12619 raft_consensus.cc:3037] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:28.545039 12619 raft_consensus.cc:504] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.545835 12619 leader_election.cc:288] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [CANDIDATE]: Term 1 election: Requested vote from peers 74c8140a399d4e25bbb671a3b5ad2530 (127.28.243.66:39289), c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905)
I20260809 22:40:28.546561 12546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46c4533e83d47ee994dbaf7e02ab3f4" candidate_uuid: "2b616e5c3bea481db474894bbdae10c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03a088f961b4a108c60321f4f806408"
I20260809 22:40:28.546547 12403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a46c4533e83d47ee994dbaf7e02ab3f4" candidate_uuid: "2b616e5c3bea481db474894bbdae10c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c8140a399d4e25bbb671a3b5ad2530"
I20260809 22:40:28.546810 12546 raft_consensus.cc:3037] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:28.546818 12403 raft_consensus.cc:3037] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:28.550386 12546 raft_consensus.cc:2454] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b616e5c3bea481db474894bbdae10c5 in term 1.
I20260809 22:40:28.550383 12403 raft_consensus.cc:2454] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b616e5c3bea481db474894bbdae10c5 in term 1.
I20260809 22:40:28.550889 12187 leader_election.cc:302] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b616e5c3bea481db474894bbdae10c5, 74c8140a399d4e25bbb671a3b5ad2530; no voters:
I20260809 22:40:28.551249 12619 raft_consensus.cc:2781] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:28.551893 12619 raft_consensus.cc:686] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 1 LEADER]: Becoming Leader. State: Replica: 2b616e5c3bea481db474894bbdae10c5, State: Running, Role: LEADER
I20260809 22:40:28.552600 12619 consensus_queue.cc:237] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } }
I20260809 22:40:28.558080 12095 catalog_manager.cc:5495] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b616e5c3bea481db474894bbdae10c5 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "2b616e5c3bea481db474894bbdae10c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } health_report { overall_health: HEALTHY } } }
I20260809 22:40:28.609632 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.602941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.28.243.65:37833)
0809 22:40:28.603220 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.603225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.603246 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:40:28.603385 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.603389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.603476 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.603636 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:40:28.604048 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.605073 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.606875 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.607356 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.607468 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.608217 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:40:28.608983 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:40:28.609122 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.609305 (+ 183us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.609385 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:40:28.619045 12403 raft_consensus.cc:1261] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 FOLLOWER]: Refusing update from remote peer 2b616e5c3bea481db474894bbdae10c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:28.619057 12546 raft_consensus.cc:1261] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Refusing update from remote peer 2b616e5c3bea481db474894bbdae10c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:28.620121 12623 consensus_queue.cc:1035] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:28.620596 12619 consensus_queue.cc:1035] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:28.633147 12626 mvcc.cc:205] Tried to move back new op lower bound from 7316747176412450816 to 7316747176155824128. Current Snapshot: MvccSnapshot[applied={T|T < 7316747176412450816}]
I20260809 22:40:28.649209 12094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:28.649559 12094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:28.653800 12094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Sending DeleteTablet for 3 replicas of tablet a46c4533e83d47ee994dbaf7e02ab3f4
I20260809 22:40:28.655162 12381 tablet_service.cc:1514] Processing DeleteTablet for tablet a46c4533e83d47ee994dbaf7e02ab3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:28 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:38498
I20260809 22:40:28.655958 12235 tablet_service.cc:1514] Processing DeleteTablet for tablet a46c4533e83d47ee994dbaf7e02ab3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:28 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:59946
I20260809 22:40:28.656221 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 12166
I20260809 22:40:28.657188 12524 tablet_service.cc:1514] Processing DeleteTablet for tablet a46c4533e83d47ee994dbaf7e02ab3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:28 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:46912
I20260809 22:40:28.666499 12638 tablet_replica.cc:330] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: stopping tablet replica
I20260809 22:40:28.667917 12638 raft_consensus.cc:2229] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:40:28.669550 12638 raft_consensus.cc:2258] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:28.675539 12640 tablet_replica.cc:330] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: stopping tablet replica
I20260809 22:40:28.676096 12640 raft_consensus.cc:2229] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:28.676666 12640 raft_consensus.cc:2258] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:28.677587 12637 tablet_replica.cc:330] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: stopping tablet replica
I20260809 22:40:28.678010 12637 raft_consensus.cc:2229] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:28.678493 12637 raft_consensus.cc:2258] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:28.679171 12638 ts_tablet_manager.cc:1905] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:28.680634 12637 ts_tablet_manager.cc:1905] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:28.686519 12640 ts_tablet_manager.cc:1905] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:28.687211 12638 ts_tablet_manager.cc:1918] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:28.687491 12638 log.cc:1196] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/a46c4533e83d47ee994dbaf7e02ab3f4
I20260809 22:40:28.688059 12637 ts_tablet_manager.cc:1918] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:28.688197 12638 ts_tablet_manager.cc:1939] T a46c4533e83d47ee994dbaf7e02ab3f4 P 2b616e5c3bea481db474894bbdae10c5: Deleting consensus metadata
I20260809 22:40:28.688314 12637 log.cc:1196] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/a46c4533e83d47ee994dbaf7e02ab3f4
I20260809 22:40:28.688947 12637 ts_tablet_manager.cc:1939] T a46c4533e83d47ee994dbaf7e02ab3f4 P 74c8140a399d4e25bbb671a3b5ad2530: Deleting consensus metadata
I20260809 22:40:28.690780 12082 catalog_manager.cc:4841] TS 2b616e5c3bea481db474894bbdae10c5 (127.28.243.65:37833): tablet a46c4533e83d47ee994dbaf7e02ab3f4 (table test-table [id=75096e45c37e40cba253ec84453617dc]) successfully deleted
Aug 09 22:40:28 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:40:28.690840 12081 catalog_manager.cc:4841] TS 74c8140a399d4e25bbb671a3b5ad2530 (127.28.243.66:39289): tablet a46c4533e83d47ee994dbaf7e02ab3f4 (table test-table [id=75096e45c37e40cba253ec84453617dc]) successfully deleted
I20260809 22:40:28.694801 12640 ts_tablet_manager.cc:1918] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:28.695003 12640 log.cc:1196] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/a46c4533e83d47ee994dbaf7e02ab3f4
I20260809 22:40:28.695518 12640 ts_tablet_manager.cc:1939] T a46c4533e83d47ee994dbaf7e02ab3f4 P c03a088f961b4a108c60321f4f806408: Deleting consensus metadata
I20260809 22:40:28.697278 12079 catalog_manager.cc:4841] TS c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905): tablet a46c4533e83d47ee994dbaf7e02ab3f4 (table test-table [id=75096e45c37e40cba253ec84453617dc]) successfully deleted
I20260809 22:40:28.699851 12611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.658160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.28.243.65:37833)
0809 22:40:28.658430 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.658440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.679406 (+ 20966us) server_negotiation.cc:407] Connection header received
0809 22:40:28.679712 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.679717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.679815 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.679909 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:28.682043 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.683297 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.685487 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.686181 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.686277 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.692106 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:28.692122 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:28.692140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:28.692226 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:28.696525 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.697510 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.697519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.697523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.697692 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.698095 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.698105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.698107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.698573 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:28.698880 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.699194 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.699464 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:40:28.700744 12303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:28.807276 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 12166
I20260809 22:40:28.829035 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 12315
Aug 09 22:40:28 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:40:28.847690 12607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.829867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.28.243.66:39289)
0809 22:40:28.830058 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.830062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.830081 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:40:28.830224 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.830228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.830294 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.830380 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:28.831984 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.832981 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.834641 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.835168 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.835306 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.840401 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:28.840419 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:28.840470 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:28.840515 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:28.844208 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.845298 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.845307 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.845309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.845468 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.846046 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.846050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.846052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.846506 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:28.846785 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.847190 (+ 405us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.847400 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:40:28.848505 12449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:28.947916 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 12315
I20260809 22:40:28.969856 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 12460
Aug 09 22:40:28 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:40:28.986375 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:28.970754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.28.243.67:40905)
0809 22:40:28.970937 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:28.970943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:28.970965 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:40:28.971078 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:28.971084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:28.971184 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:28.971298 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:28.972759 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.973494 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.974862 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:28.975253 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:28.975400 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:28.980233 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:28.980248 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:28.980264 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:28.980310 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:28.983277 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.984054 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.984061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.984064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.984225 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:28.984737 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:28.984741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:28.984743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:28.985116 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:28.985457 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:28.985793 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 22:40:28.986040 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 22:40:28.987190 12587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:29.079806 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 12460
I20260809 22:40:29.099705 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 12062
I20260809 22:40:29.101182 12132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:29.222057 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 12062
I20260809 22:40:29.249985 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33071
--webserver_interface=127.28.243.126
--webserver_port=42237
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:29.631839 12645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:29.632272 12645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:29.632489 12645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:29.641551 12645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:29.641753 12645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:29.641893 12645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:29.642023 12645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:29.642119 12645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:29.654282 12645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=42237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:29.655951 12645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:29.657718 12645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:29.690886 12652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:29.690945 12651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:29.690907 12654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:29.691988 12645 server_base.cc:1034] running on GCE node
I20260809 22:40:29.693120 12645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:29.695006 12645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:29.696255 12645 hybrid_clock.cc:648] HybridClock initialized: now 1786315229696203 us; error 35 us; skew 500 ppm
I20260809 22:40:29.696820 12645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:29 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315229, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:29.701085 12645 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:29.703105 12645 webserver.cc:460] Webserver started at http://127.28.243.126:42237/ using document root <none> and password file <none>
I20260809 22:40:29.703934 12645 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:29.704100 12645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:29.710632 12645 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:40:29.714149 12661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:29.715981 12645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:40:29.716256 12645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "219ca4c356e0408aa821cd288b4ae82d"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "26ddbf990604feaca7d4c1ebb7ea2735"
server_key_iv: "786853ed417a086e072168edd53e71f7"
server_key_version: "encryptionkey@0"
I20260809 22:40:29.717695 12645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:29.739096 12645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:29.744484 12645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:29.744886 12645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:29.766929 12645 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33071
I20260809 22:40:29.767045 12716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33071 every 8 connection(s)
I20260809 22:40:29.768884 12645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:29.774076 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 12645
I20260809 22:40:29.774978 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:37833
--local_ip_for_outbound_sockets=127.28.243.65
--tserver_master_addrs=127.28.243.126:33071
--webserver_port=33899
--webserver_interface=127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:29.777005 12717 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:40:29.780648 12717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Bootstrap starting.
I20260809 22:40:29.803802 12717 log.cc:824] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:29.819720 12717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:40:29.820465 12717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Bootstrap complete.
I20260809 22:40:29.832015 12717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:29.833066 12717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219ca4c356e0408aa821cd288b4ae82d, State: Initialized, Role: FOLLOWER
I20260809 22:40:29.833886 12717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:29.834167 12717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:40:29.834302 12717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:40:29.834508 12717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:40:29.838868 12717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:29.839490 12717 leader_election.cc:302] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219ca4c356e0408aa821cd288b4ae82d; no voters:
I20260809 22:40:29.840196 12717 leader_election.cc:288] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 22:40:29.840360 12721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:40:29.841866 12721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 LEADER]: Becoming Leader. State: Replica: 219ca4c356e0408aa821cd288b4ae82d, State: Running, Role: LEADER
I20260809 22:40:29.842583 12721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:29.842818 12717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:29.845510 12723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219ca4c356e0408aa821cd288b4ae82d. Latest consensus state: current_term: 2 leader_uuid: "219ca4c356e0408aa821cd288b4ae82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } } }
I20260809 22:40:29.845959 12723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:29.846864 12722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "219ca4c356e0408aa821cd288b4ae82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } } }
I20260809 22:40:29.847198 12722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:29.849449 12727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:29.859226 12727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:29.861397 12727 catalog_manager.cc:1259] Loaded cluster ID: 007c20da25844fba80a29406d8bc8d87
I20260809 22:40:29.861588 12727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:40:29.865767 12727 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 22:40:29.886706 12727 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 22:40:29.903894 12727 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:40:29.906020 12727 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Loaded TSK: 0
I20260809 22:40:29.906898 12741 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:40:29.907704 12727 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 22:40:30.184968 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:30.185357 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:30.185585 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:30.193533 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:30.193717 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:30.193895 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:40:30.206158 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:37833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=33899
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:30.207598 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:30.209223 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:30.226634 12747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:30.226763 12749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:30.227474 12746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:30.227610 12719 server_base.cc:1034] running on GCE node
I20260809 22:40:30.228344 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:30.229499 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:30.230834 12719 hybrid_clock.cc:648] HybridClock initialized: now 1786315230230755 us; error 66 us; skew 500 ppm
I20260809 22:40:30.231266 12719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:30 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:30.234933 12719 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:40:30.236824 12719 webserver.cc:460] Webserver started at http://127.28.243.65:33899/ using document root <none> and password file <none>
I20260809 22:40:30.237572 12719 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:30.237725 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:30.243927 12719 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260809 22:40:30.247162 12756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:30.248998 12719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:40:30.249202 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "2b616e5c3bea481db474894bbdae10c5"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "dfd588e8c3a56212a35dbf541e62b4cf"
server_key_iv: "1cb878710285492a630bfe4a700841f1"
server_key_version: "encryptionkey@0"
I20260809 22:40:30.250272 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:30.268268 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:30.275068 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:30.275414 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:30.276690 12719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:30.278334 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:30.278467 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:30.278609 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:30.278707 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:30.317855 12719 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:37833
I20260809 22:40:30.317981 12874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:37833 every 8 connection(s)
I20260809 22:40:30.319465 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:40:30.329217 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 12719
I20260809 22:40:30.330170 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:39289
--local_ip_for_outbound_sockets=127.28.243.66
--tserver_master_addrs=127.28.243.126:33071
--webserver_port=33093
--webserver_interface=127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:40:30 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:30.351948 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:30.323224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:37357)
0809 22:40:30.324193 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:30.324251 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:30.326042 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:30.327844 (+ 1802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:30.327852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:30.328505 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:30.329172 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:30.329192 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.332090 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.332100 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:30.334812 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:30.334838 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.336193 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.336205 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:30.336604 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:30.338017 (+ 1413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:30.338055 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:30.341155 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:30.345641 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:30.345652 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:30.345669 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:30.346172 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:30.346708 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:30.346713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:30.346717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:30.346916 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:30.348649 (+ 1733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:30.348661 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:30.349344 (+ 683us) client_negotiation.cc:769] Sending connection context
0809 22:40:30.349694 (+ 350us) client_negotiation.cc:240] Negotiation successful
0809 22:40:30.350351 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":410,"threads_started":1}
I20260809 22:40:30.351948 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:30.322946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:37357 (local address 127.28.243.126:33071)
0809 22:40:30.323687 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:30.323703 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:30.324407 (+ 704us) server_negotiation.cc:407] Connection header received
0809 22:40:30.326905 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:30.326919 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:30.327297 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:30.327622 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:30.329478 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.331805 (+ 2327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.335063 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.335886 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.336096 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:30.341510 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:30.341520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:30.341536 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:30.341583 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:30.345395 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:30.346356 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:30.346364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:30.346369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:30.346545 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:30.347077 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:30.347081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:30.347083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:30.348264 (+ 1181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:30.348747 (+ 483us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:30.350000 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0809 22:40:30.350624 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":337,"threads_started":1}
I20260809 22:40:30.355386 12875 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33071
I20260809 22:40:30.355756 12875 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:30.356873 12875 heartbeater.cc:511] Master 127.28.243.126:33071 requested a full tablet report, sending...
I20260809 22:40:30.359870 12680 ts_manager.cc:194] Registered new tserver with Master: 2b616e5c3bea481db474894bbdae10c5 (127.28.243.65:37833)
I20260809 22:40:30.361449 12680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:37357
W20260809 22:40:30.760075 12879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:30.760982 12879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:30.761204 12879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:30.771469 12879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:30.771692 12879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:30.771952 12879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:40:30.783537 12879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:39289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=33093
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:30.785022 12879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:30.786434 12879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:30.804368 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:30.804272 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:30.805219 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:30.806255 12879 server_base.cc:1034] running on GCE node
I20260809 22:40:30.806998 12879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:30.808240 12879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:30.809481 12879 hybrid_clock.cc:648] HybridClock initialized: now 1786315230809377 us; error 89 us; skew 500 ppm
I20260809 22:40:30.809983 12879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:30 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:30.813949 12879 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:40:30.815727 12879 webserver.cc:460] Webserver started at http://127.28.243.66:33093/ using document root <none> and password file <none>
I20260809 22:40:30.816506 12879 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:30.816759 12879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:30.823503 12879 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:40:30.826857 12894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:30.828917 12879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:40:30.829135 12879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "74c8140a399d4e25bbb671a3b5ad2530"
format_stamp: "Formatted at 2026-08-09 22:40:27 on dist-test-slave-rgb1"
server_key: "f881d00bfec72027998cc65168673a8f"
server_key_iv: "74a70782a0d5785445275d7b4e8150e3"
server_key_version: "encryptionkey@0"
I20260809 22:40:30.830185 12879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:30.852011 12879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:30.857980 12879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:30.858304 12879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:30.859887 12879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:30.861725 12879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:30.861835 12879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:30.861999 12879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:30.862088 12879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:30.906891 12879 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39289
I20260809 22:40:30.906946 13011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39289 every 8 connection(s)
I20260809 22:40:30.909421 12879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:40:30.911114 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 12879
I20260809 22:40:30.912122 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:40905
--local_ip_for_outbound_sockets=127.28.243.67
--tserver_master_addrs=127.28.243.126:33071
--webserver_port=43317
--webserver_interface=127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:40:30 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315230, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:30.945766 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:30.912979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:48751 (local address 127.28.243.126:33071)
0809 22:40:30.922052 (+ 9073us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:30.922070 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:30.922099 (+ 29us) server_negotiation.cc:407] Connection header received
0809 22:40:30.922170 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:30.922177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:30.922296 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:30.922457 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:30.925252 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.926687 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.929304 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.930267 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.930512 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:30.937966 (+ 7454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:30.937973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:30.937977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:30.938022 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:30.941965 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:30.942979 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:30.942985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:30.942987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:30.943134 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:30.943683 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:30.943692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:30.943694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:30.944067 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:30.944236 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:30.945097 (+ 861us) server_negotiation.cc:299] Negotiation successful
0809 22:40:30.945351 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8909,"thread_start_us":8793,"threads_started":1}
I20260809 22:40:30.946866 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:30.913587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:48751)
0809 22:40:30.917413 (+ 3826us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:30.917478 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:30.920758 (+ 3280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:30.922854 (+ 2096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:30.922872 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:30.923850 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:30.924874 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:30.924911 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.927028 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.927034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:30.928979 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:30.929002 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:30.930631 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:30.930641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:30.930942 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:30.932076 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:30.932131 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:30.937481 (+ 5350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:30.942188 (+ 4707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:30.942205 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:30.942222 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:30.942824 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:30.943259 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:30.943263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:30.943267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:30.943555 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:30.944204 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:30.944213 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:30.944701 (+ 488us) client_negotiation.cc:769] Sending connection context
0809 22:40:30.944991 (+ 290us) client_negotiation.cc:240] Negotiation successful
0809 22:40:30.945612 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3448,"thread_start_us":3172,"threads_started":1}
I20260809 22:40:30.949142 13012 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33071
I20260809 22:40:30.949620 13012 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:30.951043 13012 heartbeater.cc:511] Master 127.28.243.126:33071 requested a full tablet report, sending...
I20260809 22:40:30.953564 12680 ts_manager.cc:194] Registered new tserver with Master: 74c8140a399d4e25bbb671a3b5ad2530 (127.28.243.66:39289)
I20260809 22:40:30.954567 12680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:48751
I20260809 22:40:31.366243 12875 heartbeater.cc:503] Master 127.28.243.126:33071 was elected leader, sending a full tablet report...
W20260809 22:40:31.370862 13016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:31.371189 13016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:31.371290 13016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:31.379436 13016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:31.379590 13016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:31.379711 13016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:40:31.390923 13016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:40905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=43317
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:31.392314 13016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:31.393797 13016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:31.414037 13021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:31.414646 13024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:31.414651 13022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:31.415349 13016 server_base.cc:1034] running on GCE node
I20260809 22:40:31.416112 13016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:31.417533 13016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:31.418951 13016 hybrid_clock.cc:648] HybridClock initialized: now 1786315231418862 us; error 72 us; skew 500 ppm
I20260809 22:40:31.419387 13016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:31 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315231, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:31.423462 13016 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:40:31.425720 13016 webserver.cc:460] Webserver started at http://127.28.243.67:43317/ using document root <none> and password file <none>
I20260809 22:40:31.426507 13016 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:31.426688 13016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:31.433789 13016 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 22:40:31.437712 13032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:31.440088 13016 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:40:31.440500 13016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "c03a088f961b4a108c60321f4f806408"
format_stamp: "Formatted at 2026-08-09 22:40:28 on dist-test-slave-rgb1"
server_key: "57f4fb94cee384f7520b19f3b3cc8934"
server_key_iv: "8cfb59c17c18b47438eefa1d6a1add1d"
server_key_version: "encryptionkey@0"
I20260809 22:40:31.442343 13016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:31.463354 13016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:31.468954 13016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:31.469306 13016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:31.470569 13016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:31.472416 13016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:31.472733 13016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 22:40:31.473006 13016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:31.473153 13016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:31.522951 13016 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:40905
I20260809 22:40:31.523288 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:40905 every 8 connection(s)
I20260809 22:40:31.524895 13016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:40:31.527958 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 13016
Aug 09 22:40:31 dist-test-slave-rgb1 krb5kdc[12049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315231, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:31.550870 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.529381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.28.243.126:33071)
0809 22:40:31.529968 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.529974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.530814 (+ 840us) server_negotiation.cc:407] Connection header received
0809 22:40:31.531012 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.531018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.531118 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.531251 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:31.533981 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.535164 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.537620 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.538438 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.538592 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.542951 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:31.542962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:31.542971 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:31.543032 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:31.546406 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:31.547499 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:31.547505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:31.547509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:31.547711 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:31.548249 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:31.548254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:31.548257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:31.548800 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:31.549224 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.549520 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.549863 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":458,"thread_start_us":311,"threads_started":1}
I20260809 22:40:31.555970 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.528618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:37483 (local address 127.28.243.126:33071)
0809 22:40:31.529260 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.529267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.531311 (+ 2044us) server_negotiation.cc:407] Connection header received
0809 22:40:31.534581 (+ 3270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.534588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.534693 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.534833 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:31.537827 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.539080 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.542118 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.542879 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.543050 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.547676 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:31.547683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:31.547687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:31.547722 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:31.552128 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:31.553225 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:31.553230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:31.553233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:31.553377 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:31.553904 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:31.553910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:31.553918 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:31.554246 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:31.554414 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.555278 (+ 864us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.555542 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":410,"threads_started":1}
I20260809 22:40:31.557078 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.529769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:37483)
0809 22:40:31.531091 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:31.531170 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:31.534241 (+ 3071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:31.535386 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:31.535400 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:31.536516 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:31.537644 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.537670 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.539285 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.539292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.541801 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.541828 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.543454 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.543463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.543754 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.544516 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:31.544541 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:31.547334 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:31.552314 (+ 4980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:31.552326 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:31.552350 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:31.553074 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:31.553536 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:31.553540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:31.553546 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:31.553763 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:31.554388 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:31.554396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:31.554842 (+ 446us) client_negotiation.cc:769] Sending connection context
0809 22:40:31.555163 (+ 321us) client_negotiation.cc:240] Negotiation successful
0809 22:40:31.555759 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":465,"threads_started":1}
I20260809 22:40:31.559058 13153 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33071
I20260809 22:40:31.559388 13153 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:31.561398 13153 heartbeater.cc:511] Master 127.28.243.126:33071 requested a full tablet report, sending...
I20260809 22:40:31.563737 12680 ts_manager.cc:194] Registered new tserver with Master: c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905)
I20260809 22:40:31.565528 12680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:37483
I20260809 22:40:31.568663 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:40:31.578142 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.570238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.28.243.126:33071)
0809 22:40:31.570525 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.570531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.570963 (+ 432us) server_negotiation.cc:407] Connection header received
0809 22:40:31.571133 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.571137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.571321 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.571467 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:40:31.572199 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.573341 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.575062 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.575648 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.575773 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.576489 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:40:31.577378 (+ 889us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:40:31.577534 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.577784 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.577875 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260809 22:40:31.579424 12680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:40:31.589617 12680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:40:31.632813 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.609185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.28.243.65:37833)
0809 22:40:31.611766 (+ 2581us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.611780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.622248 (+ 10468us) server_negotiation.cc:407] Connection header received
0809 22:40:31.622973 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.622984 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.623309 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.623588 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:31.625006 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.626550 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.630279 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.631360 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.631572 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.631856 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.632206 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.632403 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"spinlock_wait_cycles":171264,"thread_start_us":1940,"threads_started":1}
I20260809 22:40:31.633540 13165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.608103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37833 (local address 127.0.0.1:59952)
0809 22:40:31.622030 (+ 13927us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:31.622064 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:31.622721 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:31.624062 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:31.624075 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:31.624133 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:31.624717 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.624736 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.626712 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.626718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.630038 (+ 3320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.630054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.631653 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.631660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.631922 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.631982 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 22:40:31.632139 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 22:40:31.632375 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13719,"thread_start_us":14189,"threads_started":1}
I20260809 22:40:31.639459 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.609173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.28.243.67:40905)
0809 22:40:31.623154 (+ 13981us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.623164 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.625788 (+ 2624us) server_negotiation.cc:407] Connection header received
0809 22:40:31.625909 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.625919 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.626546 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.626815 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:31.628019 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.630068 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.634212 (+ 4144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.637370 (+ 3158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.637575 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.638135 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.638881 (+ 746us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.639098 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13770,"thread_start_us":13986,"threads_started":1}
I20260809 22:40:31.639657 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.608819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:40905 (local address 127.0.0.1:46944)
0809 22:40:31.625478 (+ 16659us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:31.625512 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:31.625646 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:31.626851 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:31.626866 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:31.626902 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:31.627742 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.627755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.630271 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.630276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.633654 (+ 3378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.633703 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.638377 (+ 4674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.638385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.638593 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.638662 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 22:40:31.638776 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 22:40:31.638945 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16517,"mutex_wait_us":6420,"thread_start_us":10087,"threads_started":1}
I20260809 22:40:31.640340 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.608175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.28.243.66:39289)
0809 22:40:31.625861 (+ 17686us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.625872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.625899 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:40:31.625993 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.626134 (+ 141us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.626552 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.626827 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:31.628221 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.630068 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.635078 (+ 5010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.637972 (+ 2894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.638171 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.638344 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.639486 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.640076 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17493,"thread_start_us":17367,"threads_started":1}
I20260809 22:40:31.640683 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.607954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39289 (local address 127.0.0.1:38514)
0809 22:40:31.615421 (+ 7467us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:31.615503 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:31.615833 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:31.627062 (+ 11229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:31.627069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:31.627098 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:31.627988 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.628009 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.630270 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.630276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.634535 (+ 4259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.634564 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.638500 (+ 3936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.638506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.638683 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.638720 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 22:40:31.638828 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 22:40:31.638977 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7024,"mutex_wait_us":196,"thread_start_us":633,"threads_started":1}
I20260809 22:40:31.641232 12807 tablet_service.cc:1467] Processing CreateTablet for tablet 517b352500e242ad89e059aaa1eda0a2 (DEFAULT_TABLE table=test-table [id=2b8804afa75943f29c465f50b915e82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:31.642861 12807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517b352500e242ad89e059aaa1eda0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:31.648833 12944 tablet_service.cc:1467] Processing CreateTablet for tablet 517b352500e242ad89e059aaa1eda0a2 (DEFAULT_TABLE table=test-table [id=2b8804afa75943f29c465f50b915e82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:31.648833 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 517b352500e242ad89e059aaa1eda0a2 (DEFAULT_TABLE table=test-table [id=2b8804afa75943f29c465f50b915e82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:31.651542 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517b352500e242ad89e059aaa1eda0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:31.651542 12944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 517b352500e242ad89e059aaa1eda0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:31.661810 13170 tablet_bootstrap.cc:492] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Bootstrap starting.
I20260809 22:40:31.666391 13170 tablet_bootstrap.cc:654] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:31.667836 13170 log.cc:824] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:31.671242 13172 tablet_bootstrap.cc:492] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Bootstrap starting.
I20260809 22:40:31.671998 13170 tablet_bootstrap.cc:492] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: No bootstrap required, opened a new log
I20260809 22:40:31.672420 13170 ts_tablet_manager.cc:1397] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260809 22:40:31.672417 13173 tablet_bootstrap.cc:492] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Bootstrap starting.
I20260809 22:40:31.677067 13173 tablet_bootstrap.cc:654] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:31.678541 13173 log.cc:824] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:31.680953 13172 tablet_bootstrap.cc:654] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:31.682075 13172 log.cc:824] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:31.682717 13173 tablet_bootstrap.cc:492] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: No bootstrap required, opened a new log
I20260809 22:40:31.683038 13173 ts_tablet_manager.cc:1397] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 22:40:31.683810 13170 raft_consensus.cc:348] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.684312 13170 raft_consensus.cc:374] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:31.684420 13170 raft_consensus.cc:729] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b616e5c3bea481db474894bbdae10c5, State: Initialized, Role: FOLLOWER
I20260809 22:40:31.685365 13172 tablet_bootstrap.cc:492] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: No bootstrap required, opened a new log
I20260809 22:40:31.685164 13170 consensus_queue.cc:260] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.685670 13172 ts_tablet_manager.cc:1397] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20260809 22:40:31.690203 13170 ts_tablet_manager.cc:1428] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Time spent starting tablet: real 0.018s user 0.014s sys 0.003s
I20260809 22:40:31.694613 13173 raft_consensus.cc:348] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.694852 13172 raft_consensus.cc:348] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.695158 13173 raft_consensus.cc:374] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:31.695261 13173 raft_consensus.cc:729] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c8140a399d4e25bbb671a3b5ad2530, State: Initialized, Role: FOLLOWER
I20260809 22:40:31.695286 13172 raft_consensus.cc:374] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:31.695395 13172 raft_consensus.cc:729] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03a088f961b4a108c60321f4f806408, State: Initialized, Role: FOLLOWER
I20260809 22:40:31.696058 13172 consensus_queue.cc:260] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.696044 13173 consensus_queue.cc:260] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.697803 13012 heartbeater.cc:503] Master 127.28.243.126:33071 was elected leader, sending a full tablet report...
I20260809 22:40:31.697999 13153 heartbeater.cc:503] Master 127.28.243.126:33071 was elected leader, sending a full tablet report...
I20260809 22:40:31.698437 13173 ts_tablet_manager.cc:1428] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Time spent starting tablet: real 0.015s user 0.015s sys 0.002s
I20260809 22:40:31.698458 13172 ts_tablet_manager.cc:1428] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
W20260809 22:40:31.776829 13154 tablet.cc:2369] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:40:31.823040 12876 tablet.cc:2369] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:31.883901 13178 raft_consensus.cc:482] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:40:31.884287 13178 raft_consensus.cc:504] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.886276 13178 leader_election.cc:288] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b616e5c3bea481db474894bbdae10c5 (127.28.243.65:37833), c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905)
I20260809 22:40:31.895088 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.886838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:45871 (local address 127.28.243.65:37833)
0809 22:40:31.887129 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.887140 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.888287 (+ 1147us) server_negotiation.cc:407] Connection header received
0809 22:40:31.888495 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.888498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.888629 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.888744 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:31.889484 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.890401 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.892639 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.893498 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.893657 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.893719 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.894611 (+ 892us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.894808 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 22:40:31.895112 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.886759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37833 (local address 127.28.243.66:45871)
0809 22:40:31.888185 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:31.888201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:31.888321 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:31.888759 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:31.888764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:31.888787 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:31.889311 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.889321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.890607 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.890613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.892145 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.892158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.893792 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.893800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.894261 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.894304 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 22:40:31.894400 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 22:40:31.894539 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1304,"mutex_wait_us":58,"thread_start_us":905,"threads_started":1}
I20260809 22:40:31.895236 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.887031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:41029 (local address 127.28.243.67:40905)
0809 22:40:31.887291 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.887300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.888207 (+ 907us) server_negotiation.cc:407] Connection header received
0809 22:40:31.888476 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.888481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.888545 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.888718 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:31.889367 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.890257 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.892471 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.893800 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.894179 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.894260 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:31.894808 (+ 548us) server_negotiation.cc:299] Negotiation successful
0809 22:40:31.894977 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 22:40:31.895673 13179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.886753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:40905 (local address 127.28.243.66:41029)
0809 22:40:31.888067 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:31.888086 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:31.888293 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:31.888740 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:31.888744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:31.888771 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:31.889198 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.889209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.890435 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.890443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.892122 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:31.892136 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:31.894157 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:31.894163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:31.894579 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:31.894617 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 22:40:31.894741 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 22:40:31.894900 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":861,"threads_started":1}
I20260809 22:40:31.896905 12829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517b352500e242ad89e059aaa1eda0a2" candidate_uuid: "74c8140a399d4e25bbb671a3b5ad2530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b616e5c3bea481db474894bbdae10c5" is_pre_election: true
I20260809 22:40:31.897145 13106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517b352500e242ad89e059aaa1eda0a2" candidate_uuid: "74c8140a399d4e25bbb671a3b5ad2530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03a088f961b4a108c60321f4f806408" is_pre_election: true
I20260809 22:40:31.897573 12829 raft_consensus.cc:2454] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8140a399d4e25bbb671a3b5ad2530 in term 0.
I20260809 22:40:31.897818 13106 raft_consensus.cc:2454] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74c8140a399d4e25bbb671a3b5ad2530 in term 0.
I20260809 22:40:31.898471 12898 leader_election.cc:302] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b616e5c3bea481db474894bbdae10c5, 74c8140a399d4e25bbb671a3b5ad2530, c03a088f961b4a108c60321f4f806408; no voters:
I20260809 22:40:31.899262 13178 raft_consensus.cc:2781] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:40:31.899477 13178 raft_consensus.cc:482] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:40:31.899626 13178 raft_consensus.cc:3037] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:31.902562 13178 raft_consensus.cc:504] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
I20260809 22:40:31.903278 13178 leader_election.cc:288] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [CANDIDATE]: Term 1 election: Requested vote from peers 2b616e5c3bea481db474894bbdae10c5 (127.28.243.65:37833), c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905)
I20260809 22:40:31.904160 13106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517b352500e242ad89e059aaa1eda0a2" candidate_uuid: "74c8140a399d4e25bbb671a3b5ad2530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03a088f961b4a108c60321f4f806408"
I20260809 22:40:31.904006 12829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "517b352500e242ad89e059aaa1eda0a2" candidate_uuid: "74c8140a399d4e25bbb671a3b5ad2530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b616e5c3bea481db474894bbdae10c5"
I20260809 22:40:31.904492 13106 raft_consensus.cc:3037] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:31.905099 12829 raft_consensus.cc:3037] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:31.908486 13106 raft_consensus.cc:2454] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8140a399d4e25bbb671a3b5ad2530 in term 1.
I20260809 22:40:31.909389 12829 raft_consensus.cc:2454] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74c8140a399d4e25bbb671a3b5ad2530 in term 1.
I20260809 22:40:31.909381 12895 leader_election.cc:302] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c8140a399d4e25bbb671a3b5ad2530, c03a088f961b4a108c60321f4f806408; no voters:
I20260809 22:40:31.910243 13178 raft_consensus.cc:2781] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:31.910948 13178 raft_consensus.cc:686] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 LEADER]: Becoming Leader. State: Replica: 74c8140a399d4e25bbb671a3b5ad2530, State: Running, Role: LEADER
I20260809 22:40:31.911763 13178 consensus_queue.cc:237] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } }
W20260809 22:40:31.913690 13013 tablet.cc:2369] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:31.917860 12679 catalog_manager.cc:5495] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74c8140a399d4e25bbb671a3b5ad2530 (127.28.243.66). New cstate: current_term: 1 leader_uuid: "74c8140a399d4e25bbb671a3b5ad2530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c8140a399d4e25bbb671a3b5ad2530" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 } health_report { overall_health: UNKNOWN } } }
I20260809 22:40:32.007073 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:31.997006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.28.243.66:39289)
0809 22:40:31.997612 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:31.997627 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:31.997815 (+ 188us) server_negotiation.cc:407] Connection header received
0809 22:40:31.998201 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:31.998213 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:31.998380 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:31.998627 (+ 247us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:40:31.999595 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.000830 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.003302 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.003795 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.003943 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:32.004897 (+ 954us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:40:32.005970 (+ 1073us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:40:32.006212 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:32.006614 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 22:40:32.006742 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260809 22:40:32.018807 13106 raft_consensus.cc:1261] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Refusing update from remote peer 74c8140a399d4e25bbb671a3b5ad2530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:32.018807 12829 raft_consensus.cc:1261] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 1 FOLLOWER]: Refusing update from remote peer 74c8140a399d4e25bbb671a3b5ad2530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:32.020236 13182 consensus_queue.cc:1035] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03a088f961b4a108c60321f4f806408" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:32.020730 13178 consensus_queue.cc:1035] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b616e5c3bea481db474894bbdae10c5" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:32.033859 13186 mvcc.cc:205] Tried to move back new op lower bound from 7316747190336049152 to 7316747189916037120. Current Snapshot: MvccSnapshot[applied={T|T < 7316747190336049152}]
I20260809 22:40:32.051721 12679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:32.052073 12679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:32.056007 12679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Sending DeleteTablet for 3 replicas of tablet 517b352500e242ad89e059aaa1eda0a2
I20260809 22:40:32.057725 12944 tablet_service.cc:1514] Processing DeleteTablet for tablet 517b352500e242ad89e059aaa1eda0a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:32 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:38514
I20260809 22:40:32.057742 12807 tablet_service.cc:1514] Processing DeleteTablet for tablet 517b352500e242ad89e059aaa1eda0a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:32 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:59952
I20260809 22:40:32.058534 13083 tablet_service.cc:1514] Processing DeleteTablet for tablet 517b352500e242ad89e059aaa1eda0a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:32 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:46944
I20260809 22:40:32.062240 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 12719
I20260809 22:40:32.074652 13197 tablet_replica.cc:330] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: stopping tablet replica
I20260809 22:40:32.076643 13197 raft_consensus.cc:2229] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:32.077986 13197 raft_consensus.cc:2258] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:32.079101 13196 tablet_replica.cc:330] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: stopping tablet replica
I20260809 22:40:32.079742 13195 tablet_replica.cc:330] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: stopping tablet replica
I20260809 22:40:32.080222 13195 raft_consensus.cc:2229] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:40:32.081077 13195 raft_consensus.cc:2258] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:32.081118 13196 raft_consensus.cc:2229] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:32.082221 13196 raft_consensus.cc:2258] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:32.083472 13197 ts_tablet_manager.cc:1905] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:32.084146 13196 ts_tablet_manager.cc:1905] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:32.092810 13195 ts_tablet_manager.cc:1905] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:32.094947 13197 ts_tablet_manager.cc:1918] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:32.095364 13197 log.cc:1196] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/517b352500e242ad89e059aaa1eda0a2
I20260809 22:40:32.096217 13197 ts_tablet_manager.cc:1939] T 517b352500e242ad89e059aaa1eda0a2 P c03a088f961b4a108c60321f4f806408: Deleting consensus metadata
I20260809 22:40:32.098626 12662 catalog_manager.cc:4841] TS c03a088f961b4a108c60321f4f806408 (127.28.243.67:40905): tablet 517b352500e242ad89e059aaa1eda0a2 (table test-table [id=2b8804afa75943f29c465f50b915e82d]) successfully deleted
I20260809 22:40:32.099431 13196 ts_tablet_manager.cc:1918] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:32.102572 13195 ts_tablet_manager.cc:1918] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:32.103116 13195 log.cc:1196] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/517b352500e242ad89e059aaa1eda0a2
I20260809 22:40:32.103209 13196 log.cc:1196] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/517b352500e242ad89e059aaa1eda0a2
I20260809 22:40:32.104698 13195 ts_tablet_manager.cc:1939] T 517b352500e242ad89e059aaa1eda0a2 P 74c8140a399d4e25bbb671a3b5ad2530: Deleting consensus metadata
I20260809 22:40:32.104715 13196 ts_tablet_manager.cc:1939] T 517b352500e242ad89e059aaa1eda0a2 P 2b616e5c3bea481db474894bbdae10c5: Deleting consensus metadata
I20260809 22:40:32.107478 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:32.064464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.28.243.65:37833)
0809 22:40:32.065720 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:32.065729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:32.067686 (+ 1957us) server_negotiation.cc:407] Connection header received
0809 22:40:32.068834 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:32.068841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:32.068943 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:32.069029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:32.073282 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.074543 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.078935 (+ 4392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.079666 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.079784 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:32.091092 (+ 11308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:32.091111 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:32.091132 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:32.091194 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:32.095858 (+ 4664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:32.102084 (+ 6226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:32.102164 (+ 80us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:32.102169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:32.102525 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:32.103608 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:32.103615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:32.103619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:32.104440 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:32.105395 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:32.106700 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0809 22:40:32.107039 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260809 22:40:32.107625 12665 catalog_manager.cc:4841] TS 2b616e5c3bea481db474894bbdae10c5 (127.28.243.65:37833): tablet 517b352500e242ad89e059aaa1eda0a2 (table test-table [id=2b8804afa75943f29c465f50b915e82d]) successfully deleted
I20260809 22:40:32.108047 12664 catalog_manager.cc:4841] TS 74c8140a399d4e25bbb671a3b5ad2530 (127.28.243.66:39289): tablet 517b352500e242ad89e059aaa1eda0a2 (table test-table [id=2b8804afa75943f29c465f50b915e82d]) successfully deleted
I20260809 22:40:32.109849 12870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:32.243921 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 12719
I20260809 22:40:32.265271 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 12879
I20260809 22:40:32.289470 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:32.266514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.28.243.66:39289)
0809 22:40:32.266828 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:32.266844 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:32.266876 (+ 32us) server_negotiation.cc:407] Connection header received
0809 22:40:32.267200 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:32.267210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:32.267316 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:32.267490 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:32.269662 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.270868 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.274086 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.274821 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.274958 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:32.280865 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:32.280886 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:32.280903 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:32.280959 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:32.284890 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:32.285984 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:32.286011 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:32.286013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:32.286220 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:32.287345 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:32.287351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:32.287354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:32.287907 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:32.288196 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:32.288713 (+ 517us) server_negotiation.cc:299] Negotiation successful
0809 22:40:32.288922 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 22:40:32.290424 13007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:32.421619 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 12879
I20260809 22:40:32.446488 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 13016
I20260809 22:40:32.464267 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:32.447372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.28.243.67:40905)
0809 22:40:32.447974 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:32.447979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:32.447996 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:40:32.448060 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:32.448064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:32.448124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:32.448219 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:32.449994 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.451102 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.453179 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:32.453632 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:32.453786 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:32.457088 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:32.457109 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:32.457131 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:32.457198 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:32.460492 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:32.461763 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:32.461773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:32.461776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:32.461966 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:32.462449 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:32.462453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:32.462455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:32.462885 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:32.463213 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:32.463613 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 22:40:32.463859 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":353,"threads_started":1}
I20260809 22:40:32.465070 13148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:32.574297 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 13016
I20260809 22:40:32.593683 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 12645
I20260809 22:40:32.594952 12712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:32.714967 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 12645
I20260809 22:40:32.741734 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33071
--webserver_interface=127.28.243.126
--webserver_port=42237
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:33.160776 13204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:33.161254 13204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:33.161453 13204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:33.171547 13204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:33.172025 13204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:33.172212 13204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:33.172351 13204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:33.172451 13204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:33.184410 13204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=42237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:33.186151 13204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:33.187552 13204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:33.207526 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:33.207744 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:33.207810 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:33.210130 13204 server_base.cc:1034] running on GCE node
I20260809 22:40:33.211674 13204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:33.214340 13204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:33.215971 13204 hybrid_clock.cc:648] HybridClock initialized: now 1786315233215838 us; error 75 us; skew 500 ppm
I20260809 22:40:33.217084 13204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:33 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315233, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:33.221962 13204 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:33.223872 13204 webserver.cc:460] Webserver started at http://127.28.243.126:42237/ using document root <none> and password file <none>
I20260809 22:40:33.224814 13204 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:33.224973 13204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:33.232124 13204 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.006s
I20260809 22:40:33.237527 13220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:33.240047 13204 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260809 22:40:33.240290 13204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "219ca4c356e0408aa821cd288b4ae82d"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "26ddbf990604feaca7d4c1ebb7ea2735"
server_key_iv: "786853ed417a086e072168edd53e71f7"
server_key_version: "encryptionkey@0"
I20260809 22:40:33.241387 13204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:33.257900 13204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:33.262894 13204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:33.263164 13204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:33.288316 13204 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33071
I20260809 22:40:33.288341 13272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33071 every 8 connection(s)
I20260809 22:40:33.289980 13204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:33.297358 13273 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:40:33.298681 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 13204
I20260809 22:40:33.299556 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:37833
--local_ip_for_outbound_sockets=127.28.243.65
--tserver_master_addrs=127.28.243.126:33071
--webserver_port=33899
--webserver_interface=127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:33.300994 13273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Bootstrap starting.
I20260809 22:40:33.325373 13273 log.cc:824] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:33.343705 13273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:40:33.344338 13273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d: Bootstrap complete.
I20260809 22:40:33.357000 13273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:33.358424 13273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219ca4c356e0408aa821cd288b4ae82d, State: Initialized, Role: FOLLOWER
I20260809 22:40:33.359390 13273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:33.359674 13273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:40:33.359817 13273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:40:33.360011 13273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:40:33.365054 13273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:33.365725 13273 leader_election.cc:302] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219ca4c356e0408aa821cd288b4ae82d; no voters:
I20260809 22:40:33.366489 13273 leader_election.cc:288] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 22:40:33.366909 13277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 22:40:33.368078 13277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [term 3 LEADER]: Becoming Leader. State: Replica: 219ca4c356e0408aa821cd288b4ae82d, State: Running, Role: LEADER
I20260809 22:40:33.368826 13277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } }
I20260809 22:40:33.369267 13273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:33.372929 13278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "219ca4c356e0408aa821cd288b4ae82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } } }
I20260809 22:40:33.372969 13279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219ca4c356e0408aa821cd288b4ae82d. Latest consensus state: current_term: 3 leader_uuid: "219ca4c356e0408aa821cd288b4ae82d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219ca4c356e0408aa821cd288b4ae82d" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33071 } } }
I20260809 22:40:33.373659 13278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:33.373665 13279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219ca4c356e0408aa821cd288b4ae82d [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:33.381714 13283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:33.395022 13283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:33.397190 13283 catalog_manager.cc:1259] Loaded cluster ID: 007c20da25844fba80a29406d8bc8d87
I20260809 22:40:33.397382 13283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 22:40:33.400178 13283 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcbd50189ad google::LogMessage::Fail() at ??:0
@ 0x7fcbd501b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcbd50184de google::LogMessage::Flush() at ??:0
@ 0x7fcbd5019a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcbeaa4b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcbeaa48976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcbeaad0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcbeaad0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcbd7c64345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcbd7c6791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcbd7c67799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcbd7c42eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcbdaf1c6db start_thread at ??:0
@ 0x7fcbd3eb261f clone at ??:0
W20260809 22:40:33.764586 13275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:33.764897 13275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:33.765149 13275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:33.772758 13275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:33.772904 13275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:33.773099 13275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:40:33.783484 13275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:37833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=33899
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:33.784951 13275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:33.786370 13275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:33.805387 13316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:33.805413 13318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:33.805632 13315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:33.806825 13275 server_base.cc:1034] running on GCE node
I20260809 22:40:33.807592 13275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:33.808967 13275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:33.810230 13275 hybrid_clock.cc:648] HybridClock initialized: now 1786315233810148 us; error 70 us; skew 500 ppm
I20260809 22:40:33.810659 13275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:33 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315233, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:33.814513 13275 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:40:33.816262 13275 webserver.cc:460] Webserver started at http://127.28.243.65:33899/ using document root <none> and password file <none>
I20260809 22:40:33.817132 13275 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:33.817306 13275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:33.825212 13275 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 22:40:33.828923 13326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:33.830982 13275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:40:33.831264 13275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "2b616e5c3bea481db474894bbdae10c5"
format_stamp: "Formatted at 2026-08-09 22:40:26 on dist-test-slave-rgb1"
server_key: "dfd588e8c3a56212a35dbf541e62b4cf"
server_key_iv: "1cb878710285492a630bfe4a700841f1"
server_key_version: "encryptionkey@0"
I20260809 22:40:33.832381 13275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:33.847359 13275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:33.852352 13275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:33.852787 13275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:33.854089 13275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:33.855679 13275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:33.855832 13275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:33.855986 13275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:33.856087 13275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:33.900233 13275 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:37833
I20260809 22:40:33.900322 13447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:37833 every 8 connection(s)
I20260809 22:40:33.902030 13275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:40:33.907255 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 13275
I20260809 22:40:33.908411 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:39289
--local_ip_for_outbound_sockets=127.28.243.66
--tserver_master_addrs=127.28.243.126:33071
--webserver_port=33093
--webserver_interface=127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:40:33.908535 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:33.905506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:60883)
0809 22:40:33.906549 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:33.906889 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":380,"threads_started":1}
W20260809 22:40:33.909783 13448 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:33071 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.126:33071: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
I20260809 22:40:33.911257 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:33.910645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:56645)
0809 22:40:33.910926 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:33.911063 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260809 22:40:33.912765 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:33.912190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:37037)
0809 22:40:33.912405 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:33.912537 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260809 22:40:33.913113 13448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 22:40:34.290282 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:34.290604 13452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:34.290869 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:34.299461 13452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:34.299608 13452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:34.299893 13452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:40:34.311256 13452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:39289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=33093
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:34.312801 13452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:34.314250 13452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:34.330456 13457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:34.330591 13458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:34.330591 13461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:34.332069 13452 server_base.cc:1034] running on GCE node
I20260809 22:40:34.332959 13452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:34.334401 13452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:34.336218 13452 hybrid_clock.cc:648] HybridClock initialized: now 1786315234336090 us; error 101 us; skew 500 ppm
I20260809 22:40:34.336735 13452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:34 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315234, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:34.340761 13452 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:40:34.342701 13452 webserver.cc:460] Webserver started at http://127.28.243.66:33093/ using document root <none> and password file <none>
I20260809 22:40:34.343510 13452 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:34.343674 13452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:34.349931 13452 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:40:34.353371 13468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:34.355036 13452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:40:34.355252 13452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "74c8140a399d4e25bbb671a3b5ad2530"
format_stamp: "Formatted at 2026-08-09 22:40:27 on dist-test-slave-rgb1"
server_key: "f881d00bfec72027998cc65168673a8f"
server_key_iv: "74a70782a0d5785445275d7b4e8150e3"
server_key_version: "encryptionkey@0"
I20260809 22:40:34.356385 13452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:34.379503 13452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:34.385196 13452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:34.385541 13452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:34.386720 13452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:34.388803 13452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:34.388960 13452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:34.389106 13452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:34.389200 13452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:34.433382 13452 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39289
I20260809 22:40:34.433422 13587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39289 every 8 connection(s)
I20260809 22:40:34.435015 13452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:40:34.441140 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:34.438906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:50619)
0809 22:40:34.439726 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:34.439939 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":339,"threads_started":1}
W20260809 22:40:34.441953 13588 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:33071 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.126:33071: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
I20260809 22:40:34.443173 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:34.442752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:59281)
0809 22:40:34.442920 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:34.443060 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 22:40:34.443989 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 13452
I20260809 22:40:34.444516 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:34.444067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:48663)
0809 22:40:34.444262 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:34.444350 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260809 22:40:34.444844 13588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 22:40:34.444983 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:40905
--local_ip_for_outbound_sockets=127.28.243.67
--tserver_master_addrs=127.28.243.126:33071
--webserver_port=43317
--webserver_interface=127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:34.837190 13591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:34.837594 13591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:34.838096 13591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:34.847148 13591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:34.847292 13591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:34.847494 13591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:40:34.859684 13591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:40905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=43317
--tserver_master_addrs=127.28.243.126:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:34.861572 13591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:34.863515 13591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:34.880679 13598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:34.880752 13600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:34.880731 13597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:34.882498 13591 server_base.cc:1034] running on GCE node
I20260809 22:40:34.883495 13591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:34.884828 13591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:34.886255 13591 hybrid_clock.cc:648] HybridClock initialized: now 1786315234886216 us; error 55 us; skew 500 ppm
I20260809 22:40:34.886631 13591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:34 dist-test-slave-rgb1 krb5kdc[12049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315234, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:34.890599 13591 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:40:34.892350 13591 webserver.cc:460] Webserver started at http://127.28.243.67:43317/ using document root <none> and password file <none>
I20260809 22:40:34.893209 13591 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:34.893380 13591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:34.900079 13591 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 22:40:34.903954 13609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:34.906484 13591 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260809 22:40:34.906807 13591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "c03a088f961b4a108c60321f4f806408"
format_stamp: "Formatted at 2026-08-09 22:40:28 on dist-test-slave-rgb1"
server_key: "57f4fb94cee384f7520b19f3b3cc8934"
server_key_iv: "8cfb59c17c18b47438eefa1d6a1add1d"
server_key_version: "encryptionkey@0"
I20260809 22:40:34.908262 13591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:34.915056 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:34.913894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:60909)
0809 22:40:34.914493 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:34.914685 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":264,"threads_started":1}
I20260809 22:40:34.926980 13591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:34.932518 13591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:34.932862 13591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:34.934154 13591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:34.935812 13591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:34.935917 13591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:34.936012 13591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:34.936095 13591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:34.982960 13591 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:40905
I20260809 22:40:34.983012 13728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:40905 every 8 connection(s)
I20260809 22:40:34.985368 13591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:40:34.991628 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:34.988999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:51957)
0809 22:40:34.989858 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:34.990157 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":333,"threads_started":1}
I20260809 22:40:34.992115 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 13591
W20260809 22:40:34.992383 13729 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:33071 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.126:33071: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
I20260809 22:40:34.993384 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:34.993002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:50651)
0809 22:40:34.993180 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:34.993268 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 22:40:34.994277 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:34.993912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:54761)
0809 22:40:34.994073 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:34.994161 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 22:40:34.994525 13729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 22:40:35.446358 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:35.445470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:36253)
0809 22:40:35.445978 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:35.446119 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
I20260809 22:40:35.917225 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:35.916208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:46239)
0809 22:40:35.916928 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:35.917028 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":299,"threads_started":1}
I20260809 22:40:35.996189 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:35.995203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:51725)
0809 22:40:35.995760 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:35.995913 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":299,"threads_started":1}
I20260809 22:40:36.448479 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:36.447494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:57035)
0809 22:40:36.448189 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:36.448297 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
I20260809 22:40:36.919258 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:36.918439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:58041)
0809 22:40:36.918976 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:36.919078 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":299,"threads_started":1}
I20260809 22:40:36.998159 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:36.997351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:51825)
0809 22:40:36.997874 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:36.997966 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":265,"threads_started":1}
I20260809 22:40:37.450238 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:37.449449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:59519)
0809 22:40:37.449960 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:37.450056 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":286,"threads_started":1}
I20260809 22:40:37.921203 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:37.920292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:40953)
0809 22:40:37.920911 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:37.921003 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":335,"threads_started":1}
I20260809 22:40:38.000240 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:37.999255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:34117)
0809 22:40:37.999920 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:38.000054 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":325,"threads_started":1}
I20260809 22:40:38.452217 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:38.451302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:37615)
0809 22:40:38.451855 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:38.452002 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":285,"threads_started":1}
I20260809 22:40:38.923559 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:38.922406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:49545)
0809 22:40:38.923257 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:38.923361 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":359,"threads_started":1}
I20260809 22:40:39.002278 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:39.001273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:36465)
0809 22:40:39.001983 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:39.002088 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":462,"threads_started":1}
I20260809 22:40:39.454162 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:39.453404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:49189)
0809 22:40:39.453879 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:39.453972 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260809 22:40:39.926210 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:39.925016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:40133)
0809 22:40:39.925820 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:39.926010 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":394,"threads_started":1}
I20260809 22:40:40.006027 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:40.004153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:48963)
0809 22:40:40.005028 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:40.005139 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":616,"threads_started":1}
I20260809 22:40:40.456782 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:40.455507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:33655)
0809 22:40:40.456301 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:40.456439 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":402,"threads_started":1}
I20260809 22:40:40.929039 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:40.927718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:46215)
0809 22:40:40.928502 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:40.928720 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":351,"threads_started":1}
I20260809 22:40:41.008991 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:41.007836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:41191)
0809 22:40:41.008596 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:41.008745 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":398,"threads_started":1}
I20260809 22:40:41.460879 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:41.458712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:38273)
0809 22:40:41.459845 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:41.460089 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":883,"thread_start_us":492,"threads_started":1}
I20260809 22:40:41.931860 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:41.930423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:52117)
0809 22:40:41.931381 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:41.931566 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":576,"threads_started":1}
I20260809 22:40:42.011628 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:42.010338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:42177)
0809 22:40:42.011162 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:42.011275 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":332,"threads_started":1}
I20260809 22:40:42.463232 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:42.462291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:36273)
0809 22:40:42.462922 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:42.463032 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":378,"threads_started":1}
I20260809 22:40:42.934904 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:42.933516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:38121)
0809 22:40:42.934478 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:42.934687 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":446,"threads_started":1}
I20260809 22:40:43.014014 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:43.012943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:56219)
0809 22:40:43.013707 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:43.013807 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":497,"threads_started":1}
I20260809 22:40:43.465543 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:43.464634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:34007)
0809 22:40:43.465233 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:43.465326 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":307,"threads_started":1}
I20260809 22:40:43.936903 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:43.936087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:52219)
0809 22:40:43.936646 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:43.936736 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":286,"threads_started":1}
I20260809 22:40:44.015755 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:44.014937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:41215)
0809 22:40:44.015465 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:44.015567 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":273,"threads_started":1}
I20260809 22:40:44.467556 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:44.466729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:45059)
0809 22:40:44.467280 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:44.467381 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260809 22:40:44.938946 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:44.938033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:55529)
0809 22:40:44.938591 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:44.938710 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":273,"threads_started":1}
I20260809 22:40:45.017824 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:45.016900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:55917)
0809 22:40:45.017562 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:45.017654 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":381,"threads_started":1}
I20260809 22:40:45.469580 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:45.468709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:33047)
0809 22:40:45.469282 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:45.469383 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":339,"threads_started":1}
I20260809 22:40:45.941193 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:45.940212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:57545)
0809 22:40:45.940940 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:45.941028 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":279,"threads_started":1}
I20260809 22:40:46.019701 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:46.018918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:53013)
0809 22:40:46.019400 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:46.019500 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260809 22:40:46.472090 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:46.471127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:46251)
0809 22:40:46.471803 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:46.471890 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":453,"threads_started":1}
I20260809 22:40:46.943240 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:46.942299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:55279)
0809 22:40:46.942882 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:46.943026 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":287,"threads_started":1}
I20260809 22:40:47.021792 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:47.020904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:39831)
0809 22:40:47.021514 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:47.021611 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":357,"threads_started":1}
I20260809 22:40:47.474349 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:47.473402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:57369)
0809 22:40:47.474037 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:47.474140 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":250,"threads_started":1}
I20260809 22:40:47.945331 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:47.944347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:58689)
0809 22:40:47.944988 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:47.945086 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":355,"threads_started":1}
I20260809 22:40:48.023813 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:48.022920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:43323)
0809 22:40:48.023499 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:48.023604 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":281,"threads_started":1}
I20260809 22:40:48.476974 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:48.475806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:48287)
0809 22:40:48.476651 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:48.476765 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":481,"threads_started":1}
I20260809 22:40:48.947430 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:48.946614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:53469)
0809 22:40:48.947129 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:48.947241 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
I20260809 22:40:49.025727 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:49.024956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:59991)
0809 22:40:49.025428 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:49.025527 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260809 22:40:49.479259 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:49.478386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.66:39309)
0809 22:40:49.478942 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:49.479045 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":326,"threads_started":1}
I20260809 22:40:49.949268 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:49.948446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.65:56993)
0809 22:40:49.948969 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:49.949112 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":311,"threads_started":1}
I20260809 22:40:50.004274 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 13275
I20260809 22:40:50.028122 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:50.027157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33071 (local address 127.28.243.67:44515)
0809 22:40:50.027768 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:50.027893 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":290,"threads_started":1}
I20260809 22:40:50.028780 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:50.005621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.28.243.65:37833)
0809 22:40:50.006338 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:50.006353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:50.006401 (+ 48us) server_negotiation.cc:407] Connection header received
0809 22:40:50.009180 (+ 2779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:50.009192 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:50.009707 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:50.010077 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:50.011925 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:50.013753 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:50.015503 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:50.016321 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:50.016488 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:50.020123 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:50.020135 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:50.020160 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:50.020229 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:50.023980 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:50.025389 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:50.025397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:50.025405 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:50.025650 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:50.026275 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:50.026279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:50.026281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:50.026894 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:50.027144 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:50.027879 (+ 735us) server_negotiation.cc:299] Negotiation successful
0809 22:40:50.028399 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":366,"threads_started":1}
I20260809 22:40:50.030376 13443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:50.120705 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 13275
I20260809 22:40:50.143778 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 13452
I20260809 22:40:50.163343 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:50.145050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.28.243.66:39289)
0809 22:40:50.145680 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:50.145696 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:50.145735 (+ 39us) server_negotiation.cc:407] Connection header received
0809 22:40:50.147438 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:50.147449 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:50.147904 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:50.148319 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:50.150089 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:50.151316 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:50.152606 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:50.153348 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:50.153567 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:50.156437 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:50.156445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:50.156462 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:50.156508 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:50.159579 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:50.160294 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:50.160302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:50.160308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:50.160514 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:50.161003 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:50.161007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:50.161010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:50.161728 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:50.161963 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:50.162586 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 22:40:50.163018 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
I20260809 22:40:50.165011 13583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:50.251680 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 13452
I20260809 22:40:50.275976 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 13591
I20260809 22:40:50.296489 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:50.277114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.28.243.67:40905)
0809 22:40:50.277770 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:50.277785 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:50.277827 (+ 42us) server_negotiation.cc:407] Connection header received
0809 22:40:50.279583 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:50.279593 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:50.280044 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:50.280407 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:50.282122 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:50.283361 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:50.284843 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:50.285514 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:50.285716 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:50.288738 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:50.288747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:50.288763 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:50.288804 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:50.292287 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:50.293269 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:50.293277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:50.293284 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:50.293483 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:50.294095 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:50.294100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:50.294102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:50.294709 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:50.295104 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:50.295626 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 22:40:50.296148 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":330,"threads_started":1}
I20260809 22:40:50.298117 13723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:50.384294 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 13591
2026-08-09T22:40:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26357 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:40:50 dist-test-slave-rgb1 krb5kdc[13786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:40:50 dist-test-slave-rgb1 krb5kdc[13786](info): set up 2 sockets
Aug 09 22:40:50 dist-test-slave-rgb1 krb5kdc[13786](info): commencing operation
krb5kdc: starting...
W20260809 22:40:51.635109 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:40:51 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:40:51.962052 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41591
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:52.324827 13799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:52.325171 13799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:52.325335 13799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:52.332953 13799 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 22:40:52.333069 13799 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 22:40:52.333134 13799 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 22:40:52.333220 13799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:40:52.333314 13799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:52.333413 13799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:52.333504 13799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:40:52.333599 13799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:40:52.344271 13799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:41591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:52.345777 13799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:52.347182 13799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:52.361277 13804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:52.361191 13807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:52.361239 13805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:52.363196 13799 server_base.cc:1034] running on GCE node
I20260809 22:40:52.364114 13799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:52.365746 13799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:52.367060 13799 hybrid_clock.cc:648] HybridClock initialized: now 1786315252366991 us; error 41 us; skew 500 ppm
I20260809 22:40:52.367419 13799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:52 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315252, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:52.370998 13799 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:40:52.372819 13799 webserver.cc:460] Webserver started at http://127.28.243.126:38405/ using document root <none> and password file <none>
I20260809 22:40:52.373601 13799 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:52.373744 13799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:52.374393 13799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:52.378381 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "da55f343450447a6ae58c20481c28247"
format_stamp: "Formatted at 2026-08-09 22:40:52 on dist-test-slave-rgb1"
server_key: "d1defc97befb4da55e5642441cfb3c1e"
server_key_iv: "999830aa9bd04b40c50b778db8efa1ba"
server_key_version: "encryptionkey@0"
I20260809 22:40:52.379325 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "da55f343450447a6ae58c20481c28247"
format_stamp: "Formatted at 2026-08-09 22:40:52 on dist-test-slave-rgb1"
server_key: "d1defc97befb4da55e5642441cfb3c1e"
server_key_iv: "999830aa9bd04b40c50b778db8efa1ba"
server_key_version: "encryptionkey@0"
I20260809 22:40:52.385629 13799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 22:40:52.389793 13816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:52.391327 13799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:40:52.391631 13799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "da55f343450447a6ae58c20481c28247"
format_stamp: "Formatted at 2026-08-09 22:40:52 on dist-test-slave-rgb1"
server_key: "d1defc97befb4da55e5642441cfb3c1e"
server_key_iv: "999830aa9bd04b40c50b778db8efa1ba"
server_key_version: "encryptionkey@0"
I20260809 22:40:52.391898 13799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:52.405332 13799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:52.410180 13799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:52.410485 13799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:52.431351 13799 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:41591
I20260809 22:40:52.431377 13869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:41591 every 8 connection(s)
I20260809 22:40:52.432861 13799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:40:52.434478 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 13799
I20260809 22:40:52.434789 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:40:52.435870 29645 external_mini_cluster.cc:1428] Setting key fbf4d6bd94d1678f747c686e36d11634
I20260809 22:40:52.438740 13870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 22:40:52 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:52.455591 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Bootstrap starting.
I20260809 22:40:52.461339 13870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:52.462805 13870 log.cc:824] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:52.466672 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: No bootstrap required, opened a new log
I20260809 22:40:52.466905 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:52.438386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.28.243.126:41591)
0809 22:40:52.439387 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:52.439401 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:52.439445 (+ 44us) server_negotiation.cc:407] Connection header received
0809 22:40:52.441320 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:52.441343 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:52.441790 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:52.442161 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:52.444745 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:52.446267 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:52.448746 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:52.449550 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:52.449704 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:52.455831 (+ 6127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:52.455845 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:52.455866 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:52.455937 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:52.461442 (+ 5505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:52.462573 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:52.462581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:52.462589 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:52.462866 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:52.463468 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:52.463478 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:52.463484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:52.464119 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:52.464451 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:52.464982 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 22:40:52.465464 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":416,"threads_started":1}
I20260809 22:40:52.474735 13870 raft_consensus.cc:348] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:40:52.475306 13870 raft_consensus.cc:374] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:52.475406 13870 raft_consensus.cc:729] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da55f343450447a6ae58c20481c28247, State: Initialized, Role: FOLLOWER
I20260809 22:40:52.476064 13870 consensus_queue.cc:260] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:40:52.476300 13870 raft_consensus.cc:388] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:40:52.476420 13870 raft_consensus.cc:482] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:40:52.476621 13870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:52.478961 13870 raft_consensus.cc:504] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:40:52.479480 13870 leader_election.cc:302] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da55f343450447a6ae58c20481c28247; no voters:
I20260809 22:40:52.480228 13870 leader_election.cc:288] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:40:52.480427 13875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:52.482097 13875 raft_consensus.cc:686] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 LEADER]: Becoming Leader. State: Replica: da55f343450447a6ae58c20481c28247, State: Running, Role: LEADER
I20260809 22:40:52.482817 13870 sys_catalog.cc:564] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:40:52.482987 13875 consensus_queue.cc:237] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:40:52.486768 13877 sys_catalog.cc:455] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da55f343450447a6ae58c20481c28247. Latest consensus state: current_term: 1 leader_uuid: "da55f343450447a6ae58c20481c28247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } } }
I20260809 22:40:52.487318 13877 sys_catalog.cc:458] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:52.488083 13876 sys_catalog.cc:455] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da55f343450447a6ae58c20481c28247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } } }
I20260809 22:40:52.488454 13876 sys_catalog.cc:458] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: This master's current role is: LEADER
I20260809 22:40:52.497390 13884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:40:52.503345 13884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:40:52.514578 13884 catalog_manager.cc:1347] Generated new cluster ID: 6da536f7146f4095a7fb3757c1ce8383
I20260809 22:40:52.514737 13884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:40:52.527889 13884 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:40:52.529458 13884 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:40:52.549036 13884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Generated new TSK 0
I20260809 22:40:52.550158 13884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:40:52.641135 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41591
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:53.001539 13897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:53.001843 13897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:53.001945 13897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:53.009752 13897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:53.009893 13897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:53.010016 13897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:40:53.020316 13897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:53.021763 13897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:53.023099 13897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:53.037132 13905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:53.037257 13902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:53.037386 13903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:53.038316 13897 server_base.cc:1034] running on GCE node
I20260809 22:40:53.038980 13897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:53.039988 13897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:53.041239 13897 hybrid_clock.cc:648] HybridClock initialized: now 1786315253041163 us; error 63 us; skew 500 ppm
I20260809 22:40:53.041600 13897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:53 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315253, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:53.045042 13897 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:40:53.046686 13897 webserver.cc:460] Webserver started at http://127.28.243.65:35107/ using document root <none> and password file <none>
I20260809 22:40:53.047407 13897 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:53.047573 13897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:53.047902 13897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:53.051209 13897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "9e671bcd53514c1ab4dec8f408df65fa"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "9cf6267c200021af7e409cd6b69081ba"
server_key_iv: "346c616d5f1e52741c41765529a8752c"
server_key_version: "encryptionkey@0"
I20260809 22:40:53.052223 13897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "9e671bcd53514c1ab4dec8f408df65fa"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "9cf6267c200021af7e409cd6b69081ba"
server_key_iv: "346c616d5f1e52741c41765529a8752c"
server_key_version: "encryptionkey@0"
I20260809 22:40:53.057926 13897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260809 22:40:53.061771 13912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:53.063294 13897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260809 22:40:53.063508 13897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "9e671bcd53514c1ab4dec8f408df65fa"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "9cf6267c200021af7e409cd6b69081ba"
server_key_iv: "346c616d5f1e52741c41765529a8752c"
server_key_version: "encryptionkey@0"
I20260809 22:40:53.063746 13897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:53.081122 13897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:53.086480 13897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:53.086829 13897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:53.087868 13897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:53.089423 13897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:53.089532 13897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:53.089699 13897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:53.089777 13897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:53.129148 13897 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:36427
I20260809 22:40:53.129251 14030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:36427 every 8 connection(s)
I20260809 22:40:53.130589 13897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:40:53.133517 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 13897
I20260809 22:40:53.133797 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:40:53.134658 29645 external_mini_cluster.cc:1428] Setting key b6dc0c560a2a0b85546ab6fc9cbaab90
Aug 09 22:40:53 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315253, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:53.157060 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:53.133678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:58497 (local address 127.28.243.126:41591)
0809 22:40:53.134442 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:53.134450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:53.135727 (+ 1277us) server_negotiation.cc:407] Connection header received
0809 22:40:53.138534 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:53.138539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:53.138623 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:53.138721 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:53.141179 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.141942 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.144833 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.145589 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.145766 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:53.150159 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:53.150164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:53.150167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:53.150205 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:53.153000 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:53.154257 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:53.154262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:53.154264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:53.154406 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:53.155018 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:53.155023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:53.155025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:53.155329 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:53.155550 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:53.156390 (+ 840us) server_negotiation.cc:299] Negotiation successful
0809 22:40:53.156674 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":498,"threads_started":1}
I20260809 22:40:53.158138 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:53.134610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:58497)
0809 22:40:53.135547 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:53.135587 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:53.138153 (+ 2566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:53.139073 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:53.139090 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:53.139977 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:53.140918 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:53.140949 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.142238 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.142246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:53.144529 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:53.144550 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.145867 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.145873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:53.146178 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:53.147018 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:53.147047 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:53.149821 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:53.153181 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:53.153194 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:53.153218 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:53.154063 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:53.154537 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:53.154546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:53.154553 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:53.154849 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:53.155527 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:53.155535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:53.155946 (+ 411us) client_negotiation.cc:769] Sending connection context
0809 22:40:53.156257 (+ 311us) client_negotiation.cc:240] Negotiation successful
0809 22:40:53.156903 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":382,"threads_started":1}
I20260809 22:40:53.160171 14031 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41591
I20260809 22:40:53.160537 14031 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:53.161376 14031 heartbeater.cc:511] Master 127.28.243.126:41591 requested a full tablet report, sending...
I20260809 22:40:53.164218 13833 ts_manager.cc:194] Registered new tserver with Master: 9e671bcd53514c1ab4dec8f408df65fa (127.28.243.65:36427)
I20260809 22:40:53.166491 13833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:58497
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:40:53.214571 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41591
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:53.592128 14037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:53.592454 14037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:53.592576 14037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:40:53.600822 14037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:53.600951 14037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:53.601069 14037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:40:53.611125 14037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:53.612493 14037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:53.613893 14037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:53.628041 14043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:53.627971 14044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:53.628065 14046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:53.629302 14037 server_base.cc:1034] running on GCE node
I20260809 22:40:53.629994 14037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:53.631002 14037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:53.632292 14037 hybrid_clock.cc:648] HybridClock initialized: now 1786315253632212 us; error 56 us; skew 500 ppm
I20260809 22:40:53.632778 14037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:53 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315253, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:53.636845 14037 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:40:53.638731 14037 webserver.cc:460] Webserver started at http://127.28.243.66:36933/ using document root <none> and password file <none>
I20260809 22:40:53.639544 14037 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:53.639732 14037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:53.640111 14037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:53.643635 14037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "7a69a84ae98344149d5819e354bd52a8"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "5951b09966d22ebe5e1818980e5396b6"
server_key_iv: "8b6bc703e4eeeadf9738acd4987e2c55"
server_key_version: "encryptionkey@0"
I20260809 22:40:53.644645 14037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "7a69a84ae98344149d5819e354bd52a8"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "5951b09966d22ebe5e1818980e5396b6"
server_key_iv: "8b6bc703e4eeeadf9738acd4987e2c55"
server_key_version: "encryptionkey@0"
I20260809 22:40:53.650851 14037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:40:53.654728 14054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:53.656670 14037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:40:53.656821 14037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "7a69a84ae98344149d5819e354bd52a8"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "5951b09966d22ebe5e1818980e5396b6"
server_key_iv: "8b6bc703e4eeeadf9738acd4987e2c55"
server_key_version: "encryptionkey@0"
I20260809 22:40:53.657019 14037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:53.674321 14037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:53.679064 14037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:53.679330 14037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:53.680321 14037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:53.681751 14037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:53.681828 14037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:53.681915 14037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:53.681962 14037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:53.719679 14037 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39405
I20260809 22:40:53.719780 14177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39405 every 8 connection(s)
I20260809 22:40:53.721136 14037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:40:53.729158 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 14037
I20260809 22:40:53.729405 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:40:53.730088 29645 external_mini_cluster.cc:1428] Setting key 737b9ab34cf80494743232b22479bc9c
Aug 09 22:40:53 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315253, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:53.745664 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:53.723860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:41921 (local address 127.28.243.126:41591)
0809 22:40:53.724508 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:53.724514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:53.725620 (+ 1106us) server_negotiation.cc:407] Connection header received
0809 22:40:53.727716 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:53.727722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:53.727834 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:53.727979 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:53.730262 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.731207 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.733686 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.734127 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.734231 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:53.739091 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:53.739097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:53.739100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:53.739138 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:53.742169 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:53.743109 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:53.743114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:53.743116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:53.743250 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:53.743732 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:53.743736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:53.743738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:53.744015 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:53.744181 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:53.745017 (+ 836us) server_negotiation.cc:299] Negotiation successful
0809 22:40:53.745264 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":354,"threads_started":1}
I20260809 22:40:53.746727 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:53.724462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:41921)
0809 22:40:53.725302 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:53.725347 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:53.727398 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:53.728251 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:53.728261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:53.729091 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:53.729992 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:53.730019 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.731404 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.731410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:53.733494 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:53.733515 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:53.734392 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:53.734403 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:53.734806 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:53.735695 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:53.735734 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:53.738758 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:53.742346 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:53.742355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:53.742374 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:53.742852 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:53.743374 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:53.743380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:53.743383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:53.743595 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:53.744160 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:53.744168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:53.744545 (+ 377us) client_negotiation.cc:769] Sending connection context
0809 22:40:53.744896 (+ 351us) client_negotiation.cc:240] Negotiation successful
0809 22:40:53.745489 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":349,"threads_started":1}
I20260809 22:40:53.748374 14178 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41591
I20260809 22:40:53.748790 14178 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:53.749814 14178 heartbeater.cc:511] Master 127.28.243.126:41591 requested a full tablet report, sending...
I20260809 22:40:53.751577 13833 ts_manager.cc:194] Registered new tserver with Master: 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405)
I20260809 22:40:53.752475 13833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:41921
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:40:53.805543 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41591
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:40:54.162571 14184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:40:54.162935 14184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:40:54.163152 14184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 22:40:54.170890 14031 heartbeater.cc:503] Master 127.28.243.126:41591 was elected leader, sending a full tablet report...
W20260809 22:40:54.172695 14184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:40:54.172830 14184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:40:54.173058 14184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:40:54.183979 14184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:40:54.185738 14184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:40:54.187165 14184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:40:54.201360 14189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:54.201423 14190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:40:54.201452 14192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:40:54.203183 14184 server_base.cc:1034] running on GCE node
I20260809 22:40:54.204020 14184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:40:54.205333 14184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:40:54.206633 14184 hybrid_clock.cc:648] HybridClock initialized: now 1786315254206578 us; error 68 us; skew 500 ppm
I20260809 22:40:54.207110 14184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:40:54 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315254, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:40:54.211226 14184 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:40:54.213044 14184 webserver.cc:460] Webserver started at http://127.28.243.67:45359/ using document root <none> and password file <none>
I20260809 22:40:54.213783 14184 fs_manager.cc:359] Metadata directory not provided
I20260809 22:40:54.213904 14184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:40:54.214208 14184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:40:54.217693 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "8d6ff93a5d754321bf1785536065d58b"
format_stamp: "Formatted at 2026-08-09 22:40:54 on dist-test-slave-rgb1"
server_key: "b836d5663ae05881eb2e0627d34f4aa6"
server_key_iv: "40e47d7e9ef2e001f7e4a6c044061482"
server_key_version: "encryptionkey@0"
I20260809 22:40:54.218536 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "8d6ff93a5d754321bf1785536065d58b"
format_stamp: "Formatted at 2026-08-09 22:40:54 on dist-test-slave-rgb1"
server_key: "b836d5663ae05881eb2e0627d34f4aa6"
server_key_iv: "40e47d7e9ef2e001f7e4a6c044061482"
server_key_version: "encryptionkey@0"
I20260809 22:40:54.224885 14184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:40:54.229050 14200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:54.230762 14184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:40:54.230937 14184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "8d6ff93a5d754321bf1785536065d58b"
format_stamp: "Formatted at 2026-08-09 22:40:54 on dist-test-slave-rgb1"
server_key: "b836d5663ae05881eb2e0627d34f4aa6"
server_key_iv: "40e47d7e9ef2e001f7e4a6c044061482"
server_key_version: "encryptionkey@0"
I20260809 22:40:54.231134 14184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:40:54.243403 14184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:40:54.248147 14184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:40:54.248461 14184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:40:54.249506 14184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:40:54.250842 14184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:40:54.250931 14184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:54.251019 14184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:40:54.251065 14184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:40:54.287952 14184 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:36219
I20260809 22:40:54.287994 14323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:36219 every 8 connection(s)
I20260809 22:40:54.289345 14184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:40:54.297982 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 14184
I20260809 22:40:54.298184 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:40:54.298712 29645 external_mini_cluster.cc:1428] Setting key 921cff4c10ca72abc1042c0df965608c
Aug 09 22:40:54 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315254, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:40:54.311960 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.292171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:46443 (local address 127.28.243.126:41591)
0809 22:40:54.292742 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.292747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.293564 (+ 817us) server_negotiation.cc:407] Connection header received
0809 22:40:54.295488 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.295494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.295589 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.295720 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:54.297438 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.298666 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.300803 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.301283 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.301389 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.305735 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:54.305741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:54.305745 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:54.305781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:54.308744 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.309553 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.309558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.309561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.309706 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.310223 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.310227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.310229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.310519 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:54.310641 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.311491 (+ 850us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.311675 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":304,"threads_started":1}
I20260809 22:40:54.313223 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.292643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:46443)
0809 22:40:54.293399 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.293437 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.295212 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.296028 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.296035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.296623 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:40:54.297244 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.297263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.298943 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.298947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.300640 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.300653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.301509 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.301517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.301807 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.302785 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:40:54.302812 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:40:54.305468 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:40:54.308892 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:54.308900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:54.308915 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:54.309413 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:54.309820 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:40:54.309824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:40:54.309826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:40:54.310094 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:40:54.310649 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:40:54.310655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:40:54.311078 (+ 423us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.311342 (+ 264us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.311780 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":292,"threads_started":1}
I20260809 22:40:54.315332 14324 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41591
I20260809 22:40:54.315696 14324 heartbeater.cc:464] Registering TS with master...
I20260809 22:40:54.316504 14324 heartbeater.cc:511] Master 127.28.243.126:41591 requested a full tablet report, sending...
I20260809 22:40:54.318270 13833 ts_manager.cc:194] Registered new tserver with Master: 8d6ff93a5d754321bf1785536065d58b (127.28.243.67:36219)
I20260809 22:40:54.319124 13833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:46443
I20260809 22:40:54.328011 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:40:54.348114 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.333478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.28.243.126:41591)
0809 22:40:54.333657 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.333662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.334061 (+ 399us) server_negotiation.cc:407] Connection header received
0809 22:40:54.334240 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.334245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.334391 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.334507 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:54.336370 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.337278 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.338659 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.339085 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.339188 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.341879 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:54.341884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:54.341888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:54.341926 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:54.344921 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.345840 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.345844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.345846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.346002 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.346500 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.346505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.346508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.346914 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:54.347154 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.347550 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.347770 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 22:40:54.353202 13833 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:40:54.360262 13833 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:40:54.401213 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.375618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.28.243.67:36219)
0809 22:40:54.391415 (+ 15797us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.391425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.391462 (+ 37us) server_negotiation.cc:407] Connection header received
0809 22:40:54.391592 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.391604 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.391971 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.392228 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:54.393534 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.394962 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.398044 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.399708 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.399867 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.400243 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.400523 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.400764 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15613,"thread_start_us":15430,"threads_started":1}
I20260809 22:40:54.401183 14342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.374897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36219 (local address 127.0.0.1:59748)
0809 22:40:54.389814 (+ 14917us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.389849 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.390035 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.392276 (+ 2241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.392283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.392319 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:54.392966 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.392981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.395196 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.395202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.397865 (+ 2663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.397881 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.399951 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.399960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.400224 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.400295 (+ 71us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.400434 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.400660 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14701,"thread_start_us":16471,"threads_started":1}
I20260809 22:40:54.401233 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.374530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36427 (local address 127.0.0.1:48728)
0809 22:40:54.375798 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.375853 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.376038 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.392701 (+ 16663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.392708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.392733 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:54.393240 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.393254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.396098 (+ 2844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.396103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.398529 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.398544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.400263 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.400269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.400447 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.400474 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.400541 (+ 67us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.400671 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":4,"thread_start_us":873,"threads_started":1}
I20260809 22:40:54.402140 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.376268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39405 (local address 127.0.0.1:40138)
0809 22:40:54.391323 (+ 15055us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.391345 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.392701 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.393586 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.393591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.393623 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:54.394069 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.394082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.396157 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.396162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.398767 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.398785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.401212 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.401224 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.401473 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.401519 (+ 46us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.401612 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.401751 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14927,"mutex_wait_us":24,"thread_start_us":13898,"threads_started":1}
I20260809 22:40:54.402644 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.374878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.28.243.65:36427)
0809 22:40:54.392022 (+ 17144us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.392032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.392063 (+ 31us) server_negotiation.cc:407] Connection header received
0809 22:40:54.392170 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.392177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.392410 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.393086 (+ 676us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:54.393482 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.395922 (+ 2440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.398739 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.400018 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.401794 (+ 1776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.402063 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.402144 (+ 81us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.402320 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16949,"thread_start_us":16791,"threads_started":1}
I20260809 22:40:54.402998 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.376649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.28.243.66:39405)
0809 22:40:54.390850 (+ 14201us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.390862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.391471 (+ 609us) server_negotiation.cc:407] Connection header received
0809 22:40:54.393020 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.393030 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.393246 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.393527 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:54.394237 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.395948 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.398981 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.400646 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.400787 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.400980 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.402491 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.402633 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14031,"thread_start_us":13899,"threads_started":1}
I20260809 22:40:54.405946 14253 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c (DEFAULT_TABLE table=test-table [id=ac178b283f354af1bd87eccbf4439358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:54.406567 14107 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c (DEFAULT_TABLE table=test-table [id=ac178b283f354af1bd87eccbf4439358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:54.406569 13962 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c (DEFAULT_TABLE table=test-table [id=ac178b283f354af1bd87eccbf4439358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:40:54.407212 14253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:54.407960 13962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:54.407979 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:40:54.420073 14344 tablet_bootstrap.cc:492] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Bootstrap starting.
I20260809 22:40:54.423605 14344 tablet_bootstrap.cc:654] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:54.424307 14346 tablet_bootstrap.cc:492] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Bootstrap starting.
I20260809 22:40:54.425208 14345 tablet_bootstrap.cc:492] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Bootstrap starting.
I20260809 22:40:54.425215 14344 log.cc:824] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:54.429401 14345 tablet_bootstrap.cc:654] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:54.429479 14346 tablet_bootstrap.cc:654] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Neither blocks nor log segments found. Creating new log.
I20260809 22:40:54.431010 14345 log.cc:824] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:54.431402 14344 tablet_bootstrap.cc:492] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: No bootstrap required, opened a new log
I20260809 22:40:54.431583 14346 log.cc:824] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Log is configured to *not* fsync() on all Append() calls
I20260809 22:40:54.431788 14344 ts_tablet_manager.cc:1397] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260809 22:40:54.441606 14346 tablet_bootstrap.cc:492] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: No bootstrap required, opened a new log
I20260809 22:40:54.441603 14345 tablet_bootstrap.cc:492] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: No bootstrap required, opened a new log
I20260809 22:40:54.441944 14345 ts_tablet_manager.cc:1397] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260809 22:40:54.442003 14346 ts_tablet_manager.cc:1397] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260809 22:40:54.442886 14344 raft_consensus.cc:348] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.443493 14344 raft_consensus.cc:374] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:54.443627 14344 raft_consensus.cc:729] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6ff93a5d754321bf1785536065d58b, State: Initialized, Role: FOLLOWER
I20260809 22:40:54.444423 14344 consensus_queue.cc:260] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.453347 14345 raft_consensus.cc:348] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.453977 14345 raft_consensus.cc:374] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:54.453715 14346 raft_consensus.cc:348] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.454160 14346 raft_consensus.cc:374] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:40:54.454177 14345 raft_consensus.cc:729] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e671bcd53514c1ab4dec8f408df65fa, State: Initialized, Role: FOLLOWER
I20260809 22:40:54.454375 14346 raft_consensus.cc:729] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a69a84ae98344149d5819e354bd52a8, State: Initialized, Role: FOLLOWER
I20260809 22:40:54.455017 14345 consensus_queue.cc:260] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.455088 14346 consensus_queue.cc:260] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.463899 14345 ts_tablet_manager.cc:1428] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Time spent starting tablet: real 0.022s user 0.012s sys 0.007s
I20260809 22:40:54.464272 14178 heartbeater.cc:503] Master 127.28.243.126:41591 was elected leader, sending a full tablet report...
I20260809 22:40:54.464777 14324 heartbeater.cc:503] Master 127.28.243.126:41591 was elected leader, sending a full tablet report...
I20260809 22:40:54.465256 14346 ts_tablet_manager.cc:1428] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Time spent starting tablet: real 0.023s user 0.012s sys 0.007s
I20260809 22:40:54.465823 14344 ts_tablet_manager.cc:1428] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Time spent starting tablet: real 0.034s user 0.016s sys 0.006s
I20260809 22:40:54.464715 14350 raft_consensus.cc:482] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:40:54.466527 14350 raft_consensus.cc:504] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.468691 14350 leader_election.cc:288] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e671bcd53514c1ab4dec8f408df65fa (127.28.243.65:36427), 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405)
W20260809 22:40:54.474367 14179 tablet.cc:2369] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:54.477141 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.468925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36427 (local address 127.28.243.67:42231)
0809 22:40:54.469245 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.469268 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.469377 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.469953 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.469958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.469984 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:54.470189 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.470205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.471802 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.471808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.473968 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.473982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.476274 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.476280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.476594 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.476632 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.476723 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.476835 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260809 22:40:54.477420 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.469219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:42231 (local address 127.28.243.65:36427)
0809 22:40:54.469449 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.469455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.469479 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:40:54.469586 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.469591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.469677 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.469774 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:54.470375 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.471616 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.474841 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.476064 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.476203 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.476256 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.476785 (+ 529us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.477027 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260809 22:40:54.478636 13983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7b45ce81444336bf0daf6a9bdd5f3c" candidate_uuid: "8d6ff93a5d754321bf1785536065d58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e671bcd53514c1ab4dec8f408df65fa" is_pre_election: true
I20260809 22:40:54.479357 13983 raft_consensus.cc:2454] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6ff93a5d754321bf1785536065d58b in term 0.
I20260809 22:40:54.479632 14352 raft_consensus.cc:482] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:40:54.479874 14352 raft_consensus.cc:504] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.480372 14203 leader_election.cc:302] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6ff93a5d754321bf1785536065d58b, 9e671bcd53514c1ab4dec8f408df65fa; no voters:
I20260809 22:40:54.480969 14350 raft_consensus.cc:2781] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:40:54.481175 14350 raft_consensus.cc:482] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:40:54.481336 14350 raft_consensus.cc:3037] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:54.481870 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.469397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39405 (local address 127.28.243.67:57023)
0809 22:40:54.473421 (+ 4024us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.473444 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.473654 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.474187 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.474192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.474219 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:54.474642 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.474655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.476724 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.476729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.478914 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.478943 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.480877 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.480913 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.481141 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.481185 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.481301 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.481457 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3881,"thread_start_us":3690,"threads_started":1}
I20260809 22:40:54.481882 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.469987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:57023 (local address 127.28.243.66:39405)
0809 22:40:54.470944 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.470950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.473531 (+ 2581us) server_negotiation.cc:407] Connection header received
0809 22:40:54.473804 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.473811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.473893 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.473975 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:54.475026 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.476432 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.479205 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.480643 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.480841 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.480945 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.481410 (+ 465us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.481581 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798}
I20260809 22:40:54.482124 14352 leader_election.cc:288] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d6ff93a5d754321bf1785536065d58b (127.28.243.67:36219), 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405)
I20260809 22:40:54.483608 14128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7b45ce81444336bf0daf6a9bdd5f3c" candidate_uuid: "8d6ff93a5d754321bf1785536065d58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a69a84ae98344149d5819e354bd52a8" is_pre_election: true
I20260809 22:40:54.484385 14128 raft_consensus.cc:2454] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d6ff93a5d754321bf1785536065d58b in term 0.
I20260809 22:40:54.486578 14350 raft_consensus.cc:504] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.487352 14350 leader_election.cc:288] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [CANDIDATE]: Term 1 election: Requested vote from peers 9e671bcd53514c1ab4dec8f408df65fa (127.28.243.65:36427), 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405)
I20260809 22:40:54.487845 13983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7b45ce81444336bf0daf6a9bdd5f3c" candidate_uuid: "8d6ff93a5d754321bf1785536065d58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e671bcd53514c1ab4dec8f408df65fa"
I20260809 22:40:54.488122 13983 raft_consensus.cc:3037] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:54.488184 14128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7b45ce81444336bf0daf6a9bdd5f3c" candidate_uuid: "8d6ff93a5d754321bf1785536065d58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a69a84ae98344149d5819e354bd52a8"
I20260809 22:40:54.488404 14128 raft_consensus.cc:3037] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:40:54.492226 14128 raft_consensus.cc:2454] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6ff93a5d754321bf1785536065d58b in term 1.
I20260809 22:40:54.492226 13983 raft_consensus.cc:2454] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d6ff93a5d754321bf1785536065d58b in term 1.
I20260809 22:40:54.492967 14203 leader_election.cc:302] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a69a84ae98344149d5819e354bd52a8, 8d6ff93a5d754321bf1785536065d58b; no voters:
I20260809 22:40:54.493430 14350 raft_consensus.cc:2781] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:40:54.494059 14350 raft_consensus.cc:686] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 1 LEADER]: Becoming Leader. State: Replica: 8d6ff93a5d754321bf1785536065d58b, State: Running, Role: LEADER
I20260809 22:40:54.494740 14350 consensus_queue.cc:237] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } }
I20260809 22:40:54.495416 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.482590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:52659 (local address 127.28.243.66:39405)
0809 22:40:54.482800 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.482859 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.483684 (+ 825us) server_negotiation.cc:407] Connection header received
0809 22:40:54.484010 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.484016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.484173 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.484463 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:54.486050 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.487796 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.492150 (+ 4354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.493849 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.494033 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.494124 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.494920 (+ 796us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.495064 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 22:40:54.495431 14355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.482697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39405 (local address 127.28.243.65:52659)
0809 22:40:54.483571 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.483595 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.483806 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.484766 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.484771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.484805 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:54.485843 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.485860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.488053 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.488063 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.491837 (+ 3774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.491864 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.494593 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.494597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.494725 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.494749 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.494866 (+ 117us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.495026 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":511,"threads_started":1}
I20260809 22:40:54.496449 14128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7b45ce81444336bf0daf6a9bdd5f3c" candidate_uuid: "9e671bcd53514c1ab4dec8f408df65fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a69a84ae98344149d5819e354bd52a8" is_pre_election: true
I20260809 22:40:54.496906 14128 raft_consensus.cc:2379] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e671bcd53514c1ab4dec8f408df65fa in current term 1: Already voted for candidate 8d6ff93a5d754321bf1785536065d58b in this term.
I20260809 22:40:54.500448 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.484142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:47213 (local address 127.28.243.67:36219)
0809 22:40:54.484353 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.484359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.493431 (+ 9072us) server_negotiation.cc:407] Connection header received
0809 22:40:54.493641 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.493649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.493759 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.493859 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:40:54.494520 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.495752 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.498349 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.499344 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.499580 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.499664 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.500009 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.500151 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 22:40:54.500631 14356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.482917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36219 (local address 127.28.243.65:47213)
0809 22:40:54.493307 (+ 10390us) negotiation.cc:107] Waiting for socket to connect
0809 22:40:54.493333 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:40:54.493509 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:40:54.493878 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:40:54.493883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:40:54.493915 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:40:54.494336 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.494347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.495917 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.495922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.498175 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:40:54.498189 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.499597 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.499604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:40:54.499792 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.499824 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 22:40:54.499978 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 22:40:54.500135 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10206,"thread_start_us":10655,"threads_started":1}
I20260809 22:40:54.501044 13832 catalog_manager.cc:5495] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d6ff93a5d754321bf1785536065d58b (127.28.243.67). New cstate: current_term: 1 leader_uuid: "8d6ff93a5d754321bf1785536065d58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } health_report { overall_health: UNKNOWN } } }
I20260809 22:40:54.501603 14273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7b45ce81444336bf0daf6a9bdd5f3c" candidate_uuid: "9e671bcd53514c1ab4dec8f408df65fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6ff93a5d754321bf1785536065d58b" is_pre_election: true
I20260809 22:40:54.502780 13916 leader_election.cc:302] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e671bcd53514c1ab4dec8f408df65fa; no voters: 7a69a84ae98344149d5819e354bd52a8, 8d6ff93a5d754321bf1785536065d58b
I20260809 22:40:54.503257 14352 raft_consensus.cc:2726] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 22:40:54.541096 14325 tablet.cc:2369] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:40:54.548399 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.542501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.28.243.67:36219)
0809 22:40:54.542670 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.542674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.542798 (+ 124us) server_negotiation.cc:407] Connection header received
0809 22:40:54.542976 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.542979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.543077 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.543207 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:40:54.543712 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.544611 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.546155 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.546573 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.546680 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.547188 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:40:54.547780 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:40:54.547892 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.548084 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.548155 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 22:40:54.557440 14128 raft_consensus.cc:1261] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Refusing update from remote peer 8d6ff93a5d754321bf1785536065d58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:54.557437 13983 raft_consensus.cc:1261] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Refusing update from remote peer 8d6ff93a5d754321bf1785536065d58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:40:54.558393 14351 consensus_queue.cc:1035] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:54.558841 14350 consensus_queue.cc:1035] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:40:54.571467 14359 mvcc.cc:205] Tried to move back new op lower bound from 7316747282656067584 to 7316747282417541120. Current Snapshot: MvccSnapshot[applied={T|T < 7316747282656067584}]
I20260809 22:40:54.588615 13833 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:54.588995 13833 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:40:54.593571 13833 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Sending DeleteTablet for 3 replicas of tablet 1e7b45ce81444336bf0daf6a9bdd5f3c
I20260809 22:40:54.595078 14253 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:54 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:59748
I20260809 22:40:54.595341 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 13897
I20260809 22:40:54.595530 14107 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:54 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:40138
I20260809 22:40:54.595999 13962 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7b45ce81444336bf0daf6a9bdd5f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:40:54 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:48728
I20260809 22:40:54.597222 14370 tablet_replica.cc:330] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: stopping tablet replica
I20260809 22:40:54.597782 14370 raft_consensus.cc:2229] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:40:54.598636 14370 raft_consensus.cc:2258] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:54.601150 14370 ts_tablet_manager.cc:1905] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:54.605588 14372 tablet_replica.cc:330] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: stopping tablet replica
I20260809 22:40:54.606257 14372 raft_consensus.cc:2229] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:54.607008 14372 raft_consensus.cc:2258] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:54.609771 14371 tablet_replica.cc:330] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: stopping tablet replica
I20260809 22:40:54.610311 14371 raft_consensus.cc:2229] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:40:54.610638 14370 ts_tablet_manager.cc:1918] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:54.610904 14370 log.cc:1196] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/1e7b45ce81444336bf0daf6a9bdd5f3c
I20260809 22:40:54.610935 14371 raft_consensus.cc:2258] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:40:54.611738 14370 ts_tablet_manager.cc:1939] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 8d6ff93a5d754321bf1785536065d58b: Deleting consensus metadata
I20260809 22:40:54.613615 14371 ts_tablet_manager.cc:1905] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:54.614033 13820 catalog_manager.cc:4841] TS 8d6ff93a5d754321bf1785536065d58b (127.28.243.67:36219): tablet 1e7b45ce81444336bf0daf6a9bdd5f3c (table test-table [id=ac178b283f354af1bd87eccbf4439358]) successfully deleted
I20260809 22:40:54.618698 14372 ts_tablet_manager.cc:1905] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:40:54.622046 14371 ts_tablet_manager.cc:1918] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:54.622228 14371 log.cc:1196] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/1e7b45ce81444336bf0daf6a9bdd5f3c
Aug 09 22:40:54 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:40:54.622840 14371 ts_tablet_manager.cc:1939] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 7a69a84ae98344149d5819e354bd52a8: Deleting consensus metadata
I20260809 22:40:54.625025 13819 catalog_manager.cc:4841] TS 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405): tablet 1e7b45ce81444336bf0daf6a9bdd5f3c (table test-table [id=ac178b283f354af1bd87eccbf4439358]) successfully deleted
I20260809 22:40:54.626600 14372 ts_tablet_manager.cc:1918] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:40:54.626834 14372 log.cc:1196] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/1e7b45ce81444336bf0daf6a9bdd5f3c
I20260809 22:40:54.627511 14372 ts_tablet_manager.cc:1939] T 1e7b45ce81444336bf0daf6a9bdd5f3c P 9e671bcd53514c1ab4dec8f408df65fa: Deleting consensus metadata
I20260809 22:40:54.629369 13819 catalog_manager.cc:4841] TS 9e671bcd53514c1ab4dec8f408df65fa (127.28.243.65:36427): tablet 1e7b45ce81444336bf0daf6a9bdd5f3c (table test-table [id=ac178b283f354af1bd87eccbf4439358]) successfully deleted
I20260809 22:40:54.633471 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.598096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.28.243.65:36427)
0809 22:40:54.598356 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.598366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.609471 (+ 11105us) server_negotiation.cc:407] Connection header received
0809 22:40:54.609833 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.609839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.609960 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.610058 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:54.612655 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.613921 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.616203 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.617049 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.617201 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.624656 (+ 7455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:54.624674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:54.624693 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:54.624760 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:54.629535 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.630612 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.630624 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.630628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.630817 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.631373 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.631379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.631383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.631926 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:54.632374 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.632716 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.633064 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 22:40:54.634191 14026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:54.730305 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 13897
I20260809 22:40:54.752441 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 14037
Aug 09 22:40:54 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:40:54.775190 14343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.754408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.28.243.66:39405)
0809 22:40:54.754648 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.754655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.754828 (+ 173us) server_negotiation.cc:407] Connection header received
0809 22:40:54.755095 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.755100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.755185 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.755312 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:54.756874 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.757673 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.760211 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.760792 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.760903 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.767521 (+ 6618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:54.767539 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:54.767556 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:54.767607 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:54.771439 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.772486 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.772497 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.772507 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.772729 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.773306 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.773312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.773316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.773838 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:54.774175 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.774563 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.774830 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 22:40:54.776108 14172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:54.915854 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 14037
I20260809 22:40:54.941792 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 14184
Aug 09 22:40:54 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:40:54.963105 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:40:54.942789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.28.243.67:36219)
0809 22:40:54.943047 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 22:40:54.943052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:40:54.943069 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:40:54.943150 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:40:54.943154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:40:54.943222 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:40:54.943303 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:40:54.945274 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.946174 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.948177 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:40:54.948962 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:40:54.949087 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:40:54.954906 (+ 5819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:40:54.954925 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:40:54.954946 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:40:54.955010 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:40:54.959490 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.960584 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.960602 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.960605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.960780 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:40:54.961243 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:40:54.961254 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:40:54.961260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:40:54.961827 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:40:54.962175 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0809 22:40:54.962471 (+ 296us) server_negotiation.cc:299] Negotiation successful
0809 22:40:54.962743 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 22:40:54.964116 14318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:55.093436 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 14184
I20260809 22:40:55.117908 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 13799
I20260809 22:40:55.119410 13864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:40:55.234393 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 13799
I20260809 22:41:15.261314 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41591
--webserver_interface=127.28.243.126
--webserver_port=38405
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:41:15.639614 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:15.640000 14378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:15.640177 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:15.648087 14378 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 22:41:15.648278 14378 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 22:41:15.648384 14378 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 22:41:15.648487 14378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:41:15.648614 14378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:15.648736 14378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:15.648828 14378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:41:15.648911 14378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:41:15.660625 14378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:41591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:15.662158 14378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:15.663627 14378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:15.693267 14384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:15.693384 14385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:15.693334 14387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:15.693657 14378 server_base.cc:1034] running on GCE node
I20260809 22:41:15.694911 14378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:15.696700 14378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:15.698045 14378 hybrid_clock.cc:648] HybridClock initialized: now 1786315275697970 us; error 62 us; skew 500 ppm
I20260809 22:41:15.698500 14378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:15 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315275, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:15.708240 14378 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:41:15.710364 14378 webserver.cc:460] Webserver started at http://127.28.243.126:38405/ using document root <none> and password file <none>
I20260809 22:41:15.711243 14378 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:15.711469 14378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:15.718216 14378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:41:15.721669 14394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:15.723382 14378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260809 22:41:15.723621 14378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "da55f343450447a6ae58c20481c28247"
format_stamp: "Formatted at 2026-08-09 22:40:52 on dist-test-slave-rgb1"
server_key: "d1defc97befb4da55e5642441cfb3c1e"
server_key_iv: "999830aa9bd04b40c50b778db8efa1ba"
server_key_version: "encryptionkey@0"
I20260809 22:41:15.724936 14378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:15.743438 14378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:15.748673 14378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:15.749022 14378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:15.770787 14378 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:41591
I20260809 22:41:15.770881 14446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:41591 every 8 connection(s)
I20260809 22:41:15.772490 14378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:41:15.777210 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 14378
I20260809 22:41:15.778097 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:36427
--local_ip_for_outbound_sockets=127.28.243.65
--tserver_master_addrs=127.28.243.126:41591
--webserver_port=35107
--webserver_interface=127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:41:15.780393 14447 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:41:15.784003 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Bootstrap starting.
I20260809 22:41:15.804828 14447 log.cc:824] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Log is configured to *not* fsync() on all Append() calls
I20260809 22:41:15.818080 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:41:15.818667 14447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Bootstrap complete.
I20260809 22:41:15.828406 14447 raft_consensus.cc:348] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:15.829633 14447 raft_consensus.cc:729] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da55f343450447a6ae58c20481c28247, State: Initialized, Role: FOLLOWER
I20260809 22:41:15.830363 14447 consensus_queue.cc:260] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:15.830687 14447 raft_consensus.cc:388] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:41:15.830824 14447 raft_consensus.cc:482] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:41:15.831027 14447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 1 FOLLOWER]: Advancing to term 2
I20260809 22:41:15.835281 14447 raft_consensus.cc:504] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:15.835876 14447 leader_election.cc:302] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da55f343450447a6ae58c20481c28247; no voters:
I20260809 22:41:15.836630 14447 leader_election.cc:288] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 22:41:15.836798 14451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 22:41:15.837972 14451 raft_consensus.cc:686] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 LEADER]: Becoming Leader. State: Replica: da55f343450447a6ae58c20481c28247, State: Running, Role: LEADER
I20260809 22:41:15.838862 14447 sys_catalog.cc:564] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:41:15.838730 14451 consensus_queue.cc:237] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:15.842212 14453 sys_catalog.cc:455] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da55f343450447a6ae58c20481c28247. Latest consensus state: current_term: 2 leader_uuid: "da55f343450447a6ae58c20481c28247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } } }
I20260809 22:41:15.842758 14453 sys_catalog.cc:458] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: This master's current role is: LEADER
I20260809 22:41:15.843606 14452 sys_catalog.cc:455] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da55f343450447a6ae58c20481c28247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } } }
I20260809 22:41:15.844007 14452 sys_catalog.cc:458] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: This master's current role is: LEADER
I20260809 22:41:15.847241 14460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:41:15.858052 14460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:41:15.860273 14460 catalog_manager.cc:1259] Loaded cluster ID: 6da536f7146f4095a7fb3757c1ce8383
I20260809 22:41:15.860527 14460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:41:15.866578 14460 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:41:15.869069 14460 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Loaded TSK: 0
I20260809 22:41:15.897183 14473 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:41:15.907004 14460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Generated new TSK 1
I20260809 22:41:15.907415 14460 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 22:41:16.218787 14449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:16.219172 14449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:16.219421 14449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:16.227366 14449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:16.227550 14449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:16.227731 14449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:41:16.238415 14449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:36427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=35107
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:16.239953 14449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:16.241431 14449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:16.259341 14480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:16.262063 14479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:16.262446 14482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:16.263198 14449 server_base.cc:1034] running on GCE node
I20260809 22:41:16.264170 14449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:16.265625 14449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:16.266893 14449 hybrid_clock.cc:648] HybridClock initialized: now 1786315276266829 us; error 57 us; skew 500 ppm
I20260809 22:41:16.267418 14449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:16 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:16.272109 14449 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:41:16.274802 14449 webserver.cc:460] Webserver started at http://127.28.243.65:35107/ using document root <none> and password file <none>
I20260809 22:41:16.275897 14449 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:16.276139 14449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:16.282752 14449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:41:16.286105 14489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:16.287956 14449 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:41:16.288192 14449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "9e671bcd53514c1ab4dec8f408df65fa"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "9cf6267c200021af7e409cd6b69081ba"
server_key_iv: "346c616d5f1e52741c41765529a8752c"
server_key_version: "encryptionkey@0"
I20260809 22:41:16.290083 14449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:16.309290 14449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:16.314706 14449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:16.315050 14449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:16.316253 14449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:16.317955 14449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:16.318104 14449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:16.318264 14449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:16.318375 14449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:16.358628 14449 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:36427
I20260809 22:41:16.358731 14607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:36427 every 8 connection(s)
I20260809 22:41:16.360255 14449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:41:16.363258 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 14449
I20260809 22:41:16.364144 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:39405
--local_ip_for_outbound_sockets=127.28.243.66
--tserver_master_addrs=127.28.243.126:41591
--webserver_port=36933
--webserver_interface=127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:41:16 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:41:16.389369 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:16.363747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:52915)
0809 22:41:16.364744 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:16.364805 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:16.367677 (+ 2872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:16.369301 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:16.369314 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:16.370176 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:41:16.371215 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:16.371250 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.373159 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.373168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:16.375471 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:16.375491 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.376720 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.376727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:16.376931 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:16.377682 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:41:16.377707 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:41:16.379952 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:41:16.384836 (+ 4884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:16.384844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:16.384856 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:16.385354 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:16.385888 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:16.385892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:16.385894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:16.386137 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:16.387185 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:41:16.387191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:41:16.387498 (+ 307us) client_negotiation.cc:769] Sending connection context
0809 22:41:16.387754 (+ 256us) client_negotiation.cc:240] Negotiation successful
0809 22:41:16.388180 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":387,"threads_started":1}
I20260809 22:41:16.389820 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:16.363515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:52915 (local address 127.28.243.126:41591)
0809 22:41:16.364208 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:16.364230 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:16.364977 (+ 747us) server_negotiation.cc:407] Connection header received
0809 22:41:16.368301 (+ 3324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:16.368313 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:16.368772 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:16.369078 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:16.371493 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.372952 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.375678 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.376417 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.376714 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:16.380273 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:16.380286 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:16.380308 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:16.380373 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:16.384658 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:16.385513 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:16.385526 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:16.385536 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:16.385752 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:16.386259 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:16.386267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:16.386272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:16.387025 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:16.387218 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:16.388080 (+ 862us) server_negotiation.cc:299] Negotiation successful
0809 22:41:16.388541 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":266,"threads_started":1}
I20260809 22:41:16.392891 14608 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41591
I20260809 22:41:16.393275 14608 heartbeater.cc:464] Registering TS with master...
I20260809 22:41:16.394340 14608 heartbeater.cc:511] Master 127.28.243.126:41591 requested a full tablet report, sending...
I20260809 22:41:16.397480 14411 ts_manager.cc:194] Registered new tserver with Master: 9e671bcd53514c1ab4dec8f408df65fa (127.28.243.65:36427)
I20260809 22:41:16.399976 14411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:52915
W20260809 22:41:16.742105 14612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:16.742449 14612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:16.742630 14612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:16.750278 14612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:16.750440 14612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:16.750662 14612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:41:16.760955 14612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:39405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=36933
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:16.762336 14612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:16.763866 14612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:16.779019 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:16.779057 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:16.779641 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:16.779973 14612 server_base.cc:1034] running on GCE node
I20260809 22:41:16.780790 14612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:16.781880 14612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:16.783166 14612 hybrid_clock.cc:648] HybridClock initialized: now 1786315276783091 us; error 57 us; skew 500 ppm
I20260809 22:41:16.783551 14612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:16 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:16.787338 14612 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:41:16.789183 14612 webserver.cc:460] Webserver started at http://127.28.243.66:36933/ using document root <none> and password file <none>
I20260809 22:41:16.789897 14612 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:16.790076 14612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:16.796072 14612 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260809 22:41:16.799064 14627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:16.800864 14612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 22:41:16.801167 14612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "7a69a84ae98344149d5819e354bd52a8"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "5951b09966d22ebe5e1818980e5396b6"
server_key_iv: "8b6bc703e4eeeadf9738acd4987e2c55"
server_key_version: "encryptionkey@0"
I20260809 22:41:16.802209 14612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:16.834013 14612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:16.839430 14612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:16.839788 14612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:16.841185 14612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:16.842777 14612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:16.842871 14612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:16.842952 14612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:16.843014 14612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:16.885231 14612 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39405
I20260809 22:41:16.885282 14744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39405 every 8 connection(s)
I20260809 22:41:16.887176 14612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:41:16.893334 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 14612
I20260809 22:41:16.894112 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:36219
--local_ip_for_outbound_sockets=127.28.243.67
--tserver_master_addrs=127.28.243.126:41591
--webserver_port=45359
--webserver_interface=127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 22:41:16 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315276, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:41:16.912326 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:16.890931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:46555 (local address 127.28.243.126:41591)
0809 22:41:16.891443 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:16.891451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:16.892670 (+ 1219us) server_negotiation.cc:407] Connection header received
0809 22:41:16.894898 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:16.894904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:16.895009 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:16.895205 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:16.896869 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.897625 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.899255 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.899687 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.899828 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:16.904046 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:16.904051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:16.904055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:16.904093 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:16.908527 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:16.909546 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:16.909553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:16.909557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:16.909726 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:16.910257 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:16.910262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:16.910266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:16.910597 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:16.910787 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:16.911610 (+ 823us) server_negotiation.cc:299] Negotiation successful
0809 22:41:16.911897 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":232,"threads_started":1}
I20260809 22:41:16.913740 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:16.891600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:46555)
0809 22:41:16.892418 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:16.892465 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:16.894581 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:16.895415 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:16.895424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:16.896028 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:41:16.896686 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:16.896707 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.897794 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.897802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:16.899099 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:16.899111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:16.899942 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:16.899949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:16.900190 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:16.901034 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:41:16.901065 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:41:16.903749 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:41:16.908806 (+ 5057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:16.908815 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:16.908831 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:16.909366 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:16.909862 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:16.909868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:16.909874 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:16.910111 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:16.910747 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:41:16.910757 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:41:16.911182 (+ 425us) client_negotiation.cc:769] Sending connection context
0809 22:41:16.911487 (+ 305us) client_negotiation.cc:240] Negotiation successful
0809 22:41:16.912186 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":344,"threads_started":1}
I20260809 22:41:16.915555 14745 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41591
I20260809 22:41:16.915913 14745 heartbeater.cc:464] Registering TS with master...
I20260809 22:41:16.916917 14745 heartbeater.cc:511] Master 127.28.243.126:41591 requested a full tablet report, sending...
I20260809 22:41:16.918780 14411 ts_manager.cc:194] Registered new tserver with Master: 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405)
I20260809 22:41:16.919646 14411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:46555
W20260809 22:41:17.275774 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:17.276177 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:17.276371 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:17.284183 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:17.284351 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:17.284550 14749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:41:17.295198 14749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:36219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=45359
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:17.296663 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:17.297988 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:17.312743 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:17.312793 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:17.312844 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:17.313850 14749 server_base.cc:1034] running on GCE node
I20260809 22:41:17.314610 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:17.315683 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:17.317047 14749 hybrid_clock.cc:648] HybridClock initialized: now 1786315277316996 us; error 57 us; skew 500 ppm
I20260809 22:41:17.317481 14749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:17 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315277, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:17.321416 14749 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:41:17.323145 14749 webserver.cc:460] Webserver started at http://127.28.243.67:45359/ using document root <none> and password file <none>
I20260809 22:41:17.323899 14749 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:17.324072 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:17.330397 14749 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 22:41:17.333598 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:17.335183 14749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 22:41:17.335392 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "8d6ff93a5d754321bf1785536065d58b"
format_stamp: "Formatted at 2026-08-09 22:40:54 on dist-test-slave-rgb1"
server_key: "b836d5663ae05881eb2e0627d34f4aa6"
server_key_iv: "40e47d7e9ef2e001f7e4a6c044061482"
server_key_version: "encryptionkey@0"
I20260809 22:41:17.336531 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:17.362020 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:17.366948 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:17.367297 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:17.368419 14749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:17.369953 14749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:17.370101 14749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:17.370245 14749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:17.370337 14749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:17.404470 14608 heartbeater.cc:503] Master 127.28.243.126:41591 was elected leader, sending a full tablet report...
I20260809 22:41:17.412966 14749 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:36219
I20260809 22:41:17.413125 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:36219 every 8 connection(s)
I20260809 22:41:17.414494 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:41:17.421456 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 14749
Aug 09 22:41:17 dist-test-slave-rgb1 krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315277, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:41:17.441582 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.422633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.28.243.126:41591)
0809 22:41:17.423225 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.423232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.424137 (+ 905us) server_negotiation.cc:407] Connection header received
0809 22:41:17.424231 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.424242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.424348 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.424470 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:17.426847 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.427699 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.429697 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.430316 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.430415 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.433723 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:17.433730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:17.433734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:17.433785 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:17.437774 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.438809 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.438813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.438816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.439078 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.439606 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.439613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.439617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.440095 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:17.440418 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.440858 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.441131 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":456,"spinlock_wait_cycles":10752,"thread_start_us":295,"threads_started":1}
I20260809 22:41:17.442111 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.417398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:45263 (local address 127.28.243.126:41591)
0809 22:41:17.418070 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.418076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.418999 (+ 923us) server_negotiation.cc:407] Connection header received
0809 22:41:17.421335 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.421341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.421446 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.421603 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:17.424143 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.425212 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.427545 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.428149 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.428243 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.433316 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:17.433321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:17.433324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:17.433359 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:17.437693 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.438791 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.438798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.438801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.439022 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.439606 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.439613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.439617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.440095 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:17.440278 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.441302 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.441525 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":396,"spinlock_wait_cycles":4224,"thread_start_us":276,"threads_started":1}
I20260809 22:41:17.443194 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.418025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:45263)
0809 22:41:17.418799 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:17.418839 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:17.420773 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:17.421910 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:17.421924 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:17.422640 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:41:17.423807 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.423845 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.425400 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.425407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.427377 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.427392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.428421 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.428471 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.428834 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.429754 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:41:17.429786 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:41:17.433051 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:41:17.437891 (+ 4840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:17.437901 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:17.437929 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:17.438668 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:17.439170 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:17.439175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:17.439178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:17.439466 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:17.440267 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:41:17.440276 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:41:17.440790 (+ 514us) client_negotiation.cc:769] Sending connection context
0809 22:41:17.441213 (+ 423us) client_negotiation.cc:240] Negotiation successful
0809 22:41:17.441829 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":289,"threads_started":1}
I20260809 22:41:17.444792 14884 heartbeater.cc:347] Connected to a master server at 127.28.243.126:41591
I20260809 22:41:17.445183 14884 heartbeater.cc:464] Registering TS with master...
I20260809 22:41:17.445989 14884 heartbeater.cc:511] Master 127.28.243.126:41591 requested a full tablet report, sending...
I20260809 22:41:17.447576 14411 ts_manager.cc:194] Registered new tserver with Master: 8d6ff93a5d754321bf1785536065d58b (127.28.243.67:36219)
I20260809 22:41:17.448544 14411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:45263
I20260809 22:41:17.457690 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:41:17.466084 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.458962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.28.243.126:41591)
0809 22:41:17.459309 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.459361 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.459662 (+ 301us) server_negotiation.cc:407] Connection header received
0809 22:41:17.460008 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.460012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.460212 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.460396 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:41:17.461008 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.461813 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.463557 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.464074 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.464232 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.464928 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:41:17.465248 (+ 320us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 22:41:17.465732 (+ 484us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38756: token expired
Metrics: {"server-negotiator.queue_time_us":145}
W20260809 22:41:17.466261 14887 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38756: token expired
W20260809 22:41:17.466807 14330 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.243.126:41591)
I20260809 22:41:17.482049 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.467509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.28.243.126:41591)
0809 22:41:17.467717 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.467723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.467742 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:41:17.467933 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.467938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.468005 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.468116 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:17.469812 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.470561 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.472011 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.472443 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.472652 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.475842 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:17.475848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:17.475851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:17.475886 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:17.478810 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.479751 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.479762 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.479764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.479909 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.480355 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.480364 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.480367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.480791 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:17.481100 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.481437 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.481696 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 22:41:17.486117 14411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:41:17.494287 14411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:41:17.524335 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.508407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:36427 (local address 127.0.0.1:47376)
0809 22:41:17.509821 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:17.509884 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:17.510081 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:17.511436 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:17.511448 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:17.511505 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:17.512178 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.512196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.515162 (+ 2966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.515172 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.518942 (+ 3770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.518964 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.522692 (+ 3728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.522706 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.523111 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.523209 (+ 98us) client_negotiation.cc:769] Sending connection context
0809 22:41:17.523392 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 22:41:17.523718 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":962,"threads_started":1}
I20260809 22:41:17.525966 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.508543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.28.243.65:36427)
0809 22:41:17.509333 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.509344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.510292 (+ 948us) server_negotiation.cc:407] Connection header received
0809 22:41:17.510416 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.510428 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.510764 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.511031 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:17.512488 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.514879 (+ 2391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.519224 (+ 4345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.521777 (+ 2553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.522031 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.522427 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.525130 (+ 2703us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.525424 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":337,"threads_started":1}
I20260809 22:41:17.531677 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 2c949f1361354368bb56a0f37b6076e3 (DEFAULT_TABLE table=test-table [id=d33a98e4bb2d41b8a6acc01b42bcd92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:41:17.533389 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c949f1361354368bb56a0f37b6076e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:41:17.536494 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.510290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39405 (local address 127.0.0.1:44562)
0809 22:41:17.524689 (+ 14399us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:17.524710 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:17.524873 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:17.528339 (+ 3466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:17.528348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:17.528394 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:17.528690 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.528709 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.531331 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.531340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.533550 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.533575 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.535537 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.535546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.535767 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.535800 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 22:41:17.535908 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 22:41:17.536068 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14294,"mutex_wait_us":28,"spinlock_wait_cycles":896,"thread_start_us":16340,"threads_started":1}
I20260809 22:41:17.536242 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.509024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36219 (local address 127.0.0.1:37124)
0809 22:41:17.519013 (+ 9989us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:17.519039 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:17.519181 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:17.528138 (+ 8957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:17.528143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:17.528173 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:17.528803 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.528815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.530693 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.530699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.533699 (+ 3000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.533714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.535196 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.535203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.535423 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.535468 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 22:41:17.535565 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 22:41:17.535742 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9811,"mutex_wait_us":1,"spinlock_wait_cycles":20736,"thread_start_us":9627,"threads_started":1}
I20260809 22:41:17.536988 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.510642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.28.243.66:39405)
0809 22:41:17.527606 (+ 16964us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.527616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.527650 (+ 34us) server_negotiation.cc:407] Connection header received
0809 22:41:17.527764 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.527774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.528063 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.528916 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:17.529170 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.531114 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.533780 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.535361 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.535528 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.535755 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.536365 (+ 610us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.536612 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15928,"thread_start_us":16636,"threads_started":1}
I20260809 22:41:17.538090 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.509721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37124 (local address 127.28.243.67:36219)
0809 22:41:17.527420 (+ 17699us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.527429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.527467 (+ 38us) server_negotiation.cc:407] Connection header received
0809 22:41:17.527579 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.527587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.527888 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.528132 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:17.529007 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.530490 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.533848 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.534944 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.535127 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.535378 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.535935 (+ 557us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.536118 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17497,"thread_start_us":17291,"threads_started":1}
I20260809 22:41:17.541810 14676 tablet_service.cc:1467] Processing CreateTablet for tablet 2c949f1361354368bb56a0f37b6076e3 (DEFAULT_TABLE table=test-table [id=d33a98e4bb2d41b8a6acc01b42bcd92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:41:17.541822 14814 tablet_service.cc:1467] Processing CreateTablet for tablet 2c949f1361354368bb56a0f37b6076e3 (DEFAULT_TABLE table=test-table [id=d33a98e4bb2d41b8a6acc01b42bcd92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:41:17.543454 14814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c949f1361354368bb56a0f37b6076e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:41:17.543694 14676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c949f1361354368bb56a0f37b6076e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:41:17.548813 14902 tablet_bootstrap.cc:492] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Bootstrap starting.
I20260809 22:41:17.553319 14902 tablet_bootstrap.cc:654] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Neither blocks nor log segments found. Creating new log.
I20260809 22:41:17.554462 14902 log.cc:824] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Log is configured to *not* fsync() on all Append() calls
I20260809 22:41:17.557451 14902 tablet_bootstrap.cc:492] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: No bootstrap required, opened a new log
I20260809 22:41:17.557778 14902 ts_tablet_manager.cc:1397] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 22:41:17.560828 14904 tablet_bootstrap.cc:492] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Bootstrap starting.
I20260809 22:41:17.565050 14904 tablet_bootstrap.cc:654] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Neither blocks nor log segments found. Creating new log.
I20260809 22:41:17.565407 14905 tablet_bootstrap.cc:492] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Bootstrap starting.
I20260809 22:41:17.566754 14904 log.cc:824] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Log is configured to *not* fsync() on all Append() calls
I20260809 22:41:17.568917 14905 tablet_bootstrap.cc:654] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Neither blocks nor log segments found. Creating new log.
I20260809 22:41:17.570081 14905 log.cc:824] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Log is configured to *not* fsync() on all Append() calls
I20260809 22:41:17.570386 14904 tablet_bootstrap.cc:492] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: No bootstrap required, opened a new log
I20260809 22:41:17.570701 14904 ts_tablet_manager.cc:1397] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260809 22:41:17.571035 14902 raft_consensus.cc:348] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.571905 14902 raft_consensus.cc:374] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:41:17.572043 14902 raft_consensus.cc:729] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e671bcd53514c1ab4dec8f408df65fa, State: Initialized, Role: FOLLOWER
I20260809 22:41:17.572901 14902 consensus_queue.cc:260] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.576468 14902 ts_tablet_manager.cc:1428] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260809 22:41:17.579706 14905 tablet_bootstrap.cc:492] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: No bootstrap required, opened a new log
I20260809 22:41:17.580052 14905 ts_tablet_manager.cc:1397] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260809 22:41:17.582033 14904 raft_consensus.cc:348] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.582448 14904 raft_consensus.cc:374] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:41:17.582566 14904 raft_consensus.cc:729] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a69a84ae98344149d5819e354bd52a8, State: Initialized, Role: FOLLOWER
I20260809 22:41:17.583146 14904 consensus_queue.cc:260] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.584359 14745 heartbeater.cc:503] Master 127.28.243.126:41591 was elected leader, sending a full tablet report...
I20260809 22:41:17.585260 14904 ts_tablet_manager.cc:1428] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260809 22:41:17.588974 14905 raft_consensus.cc:348] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.589416 14905 raft_consensus.cc:374] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:41:17.589494 14905 raft_consensus.cc:729] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6ff93a5d754321bf1785536065d58b, State: Initialized, Role: FOLLOWER
I20260809 22:41:17.589975 14905 consensus_queue.cc:260] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.590982 14884 heartbeater.cc:503] Master 127.28.243.126:41591 was elected leader, sending a full tablet report...
I20260809 22:41:17.591578 14905 ts_tablet_manager.cc:1428] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
W20260809 22:41:17.613030 14609 tablet.cc:2369] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:41:17.640199 14746 tablet.cc:2369] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:41:17.666404 14886 tablet.cc:2369] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:41:17.757860 14909 raft_consensus.cc:482] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:41:17.758270 14909 raft_consensus.cc:504] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.760505 14909 leader_election.cc:288] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405), 8d6ff93a5d754321bf1785536065d58b (127.28.243.67:36219)
I20260809 22:41:17.768607 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.761239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:40807 (local address 127.28.243.67:36219)
0809 22:41:17.761544 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.761555 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.762301 (+ 746us) server_negotiation.cc:407] Connection header received
0809 22:41:17.762545 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.762549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.762612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.762727 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:17.763479 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.764226 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.766485 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.767404 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.767647 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.767743 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.768111 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.768253 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260809 22:41:17.768636 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.760998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:34969 (local address 127.28.243.66:39405)
0809 22:41:17.761248 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.761254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.762181 (+ 927us) server_negotiation.cc:407] Connection header received
0809 22:41:17.762443 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.762448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.762517 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.762627 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:17.763316 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.764117 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.766502 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.767406 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.767642 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.767745 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.768119 (+ 374us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.768266 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 22:41:17.768839 14913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.761215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:36219 (local address 127.28.243.65:40807)
0809 22:41:17.762181 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:17.762197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:17.762368 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:17.762748 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:17.762753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:17.762797 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:17.763284 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.763294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.764391 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.764397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.766293 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.766308 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.767680 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.767688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.767886 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.767945 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 22:41:17.768068 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 22:41:17.768219 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":49,"thread_start_us":602,"threads_started":1}
I20260809 22:41:17.769282 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.761075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:39405 (local address 127.28.243.65:34969)
0809 22:41:17.762058 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:17.762081 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:17.762264 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:17.762646 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:17.762650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:17.762679 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:17.763149 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.763161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.764269 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.764275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.766296 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:17.766310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.767744 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.767753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:17.767941 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.767967 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 22:41:17.768088 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 22:41:17.768235 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":32,"thread_start_us":554,"threads_started":1}
I20260809 22:41:17.770404 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c949f1361354368bb56a0f37b6076e3" candidate_uuid: "9e671bcd53514c1ab4dec8f408df65fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6ff93a5d754321bf1785536065d58b" is_pre_election: true
I20260809 22:41:17.770653 14696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c949f1361354368bb56a0f37b6076e3" candidate_uuid: "9e671bcd53514c1ab4dec8f408df65fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a69a84ae98344149d5819e354bd52a8" is_pre_election: true
I20260809 22:41:17.771018 14836 raft_consensus.cc:2454] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e671bcd53514c1ab4dec8f408df65fa in term 0.
I20260809 22:41:17.771241 14696 raft_consensus.cc:2454] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e671bcd53514c1ab4dec8f408df65fa in term 0.
I20260809 22:41:17.771888 14493 leader_election.cc:302] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6ff93a5d754321bf1785536065d58b, 9e671bcd53514c1ab4dec8f408df65fa; no voters:
I20260809 22:41:17.772362 14909 raft_consensus.cc:2781] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:41:17.772539 14909 raft_consensus.cc:482] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:41:17.772722 14909 raft_consensus.cc:3037] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:41:17.776064 14909 raft_consensus.cc:504] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.776961 14909 leader_election.cc:288] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [CANDIDATE]: Term 1 election: Requested vote from peers 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405), 8d6ff93a5d754321bf1785536065d58b (127.28.243.67:36219)
I20260809 22:41:17.777544 14696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c949f1361354368bb56a0f37b6076e3" candidate_uuid: "9e671bcd53514c1ab4dec8f408df65fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a69a84ae98344149d5819e354bd52a8"
I20260809 22:41:17.777632 14836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c949f1361354368bb56a0f37b6076e3" candidate_uuid: "9e671bcd53514c1ab4dec8f408df65fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6ff93a5d754321bf1785536065d58b"
I20260809 22:41:17.777817 14696 raft_consensus.cc:3037] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:41:17.777861 14836 raft_consensus.cc:3037] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:41:17.780386 14696 raft_consensus.cc:2454] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e671bcd53514c1ab4dec8f408df65fa in term 1.
I20260809 22:41:17.781044 14492 leader_election.cc:302] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a69a84ae98344149d5819e354bd52a8, 9e671bcd53514c1ab4dec8f408df65fa; no voters:
I20260809 22:41:17.781164 14836 raft_consensus.cc:2454] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e671bcd53514c1ab4dec8f408df65fa in term 1.
I20260809 22:41:17.781476 14909 raft_consensus.cc:2781] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:41:17.782095 14909 raft_consensus.cc:686] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 1 LEADER]: Becoming Leader. State: Replica: 9e671bcd53514c1ab4dec8f408df65fa, State: Running, Role: LEADER
I20260809 22:41:17.783051 14909 consensus_queue.cc:237] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } }
I20260809 22:41:17.788371 14411 catalog_manager.cc:5495] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e671bcd53514c1ab4dec8f408df65fa (127.28.243.65). New cstate: current_term: 1 leader_uuid: "9e671bcd53514c1ab4dec8f408df65fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e671bcd53514c1ab4dec8f408df65fa" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 36427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20260809 22:41:17.831543 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.825645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.28.243.65:36427)
0809 22:41:17.825979 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.825986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.826009 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:41:17.826268 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.826272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.826364 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.826535 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:41:17.826936 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.827673 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.829233 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.829646 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.829819 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.830398 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:41:17.830950 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:41:17.831116 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.831178 (+ 62us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.831262 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 22:41:17.841202 14696 raft_consensus.cc:1261] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Refusing update from remote peer 9e671bcd53514c1ab4dec8f408df65fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:41:17.841324 14836 raft_consensus.cc:1261] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 1 FOLLOWER]: Refusing update from remote peer 9e671bcd53514c1ab4dec8f408df65fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:41:17.842418 14909 consensus_queue.cc:1035] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6ff93a5d754321bf1785536065d58b" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:41:17.842752 14914 consensus_queue.cc:1035] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a69a84ae98344149d5819e354bd52a8" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:41:17.855252 14918 mvcc.cc:205] Tried to move back new op lower bound from 7316747378025181184 to 7316747377803558912. Current Snapshot: MvccSnapshot[applied={T|T < 7316747378025181184}]
I20260809 22:41:17.872298 14410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:41:17.872707 14410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:41:17.876802 14410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Sending DeleteTablet for 3 replicas of tablet 2c949f1361354368bb56a0f37b6076e3
I20260809 22:41:17.878263 14814 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c949f1361354368bb56a0f37b6076e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:41:17 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:37124
I20260809 22:41:17.878283 14676 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c949f1361354368bb56a0f37b6076e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:41:17 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:44562
I20260809 22:41:17.878458 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 14449
I20260809 22:41:17.879509 14931 tablet_replica.cc:330] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: stopping tablet replica
I20260809 22:41:17.879532 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c949f1361354368bb56a0f37b6076e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:41:17 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:47376
I20260809 22:41:17.879969 14931 raft_consensus.cc:2229] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:41:17.880517 14931 raft_consensus.cc:2258] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:41:17.881341 14930 tablet_replica.cc:330] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: stopping tablet replica
I20260809 22:41:17.881734 14930 raft_consensus.cc:2229] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:41:17.882230 14930 raft_consensus.cc:2258] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:41:17.885025 14932 tablet_replica.cc:330] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: stopping tablet replica
I20260809 22:41:17.885447 14932 raft_consensus.cc:2229] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:41:17.886176 14932 raft_consensus.cc:2258] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:41:17.886494 14930 ts_tablet_manager.cc:1905] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:41:17.887627 14931 ts_tablet_manager.cc:1905] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:41:17.893988 14930 ts_tablet_manager.cc:1918] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:41:17.894299 14930 log.cc:1196] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/2c949f1361354368bb56a0f37b6076e3
I20260809 22:41:17.896240 14930 ts_tablet_manager.cc:1939] T 2c949f1361354368bb56a0f37b6076e3 P 8d6ff93a5d754321bf1785536065d58b: Deleting consensus metadata
I20260809 22:41:17.898339 14398 catalog_manager.cc:4841] TS 8d6ff93a5d754321bf1785536065d58b (127.28.243.67:36219): tablet 2c949f1361354368bb56a0f37b6076e3 (table test-table [id=d33a98e4bb2d41b8a6acc01b42bcd92e]) successfully deleted
I20260809 22:41:17.899466 14931 ts_tablet_manager.cc:1918] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:41:17.899662 14931 log.cc:1196] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/2c949f1361354368bb56a0f37b6076e3
I20260809 22:41:17.900082 14932 ts_tablet_manager.cc:1905] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:41:17.900228 14931 ts_tablet_manager.cc:1939] T 2c949f1361354368bb56a0f37b6076e3 P 7a69a84ae98344149d5819e354bd52a8: Deleting consensus metadata
I20260809 22:41:17.908365 14932 ts_tablet_manager.cc:1918] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:41:17.908659 14932 log.cc:1196] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/2c949f1361354368bb56a0f37b6076e3
I20260809 22:41:17.908615 14397 catalog_manager.cc:4841] TS 7a69a84ae98344149d5819e354bd52a8 (127.28.243.66:39405): tablet 2c949f1361354368bb56a0f37b6076e3 (table test-table [id=d33a98e4bb2d41b8a6acc01b42bcd92e]) successfully deleted
I20260809 22:41:17.909292 14932 ts_tablet_manager.cc:1939] T 2c949f1361354368bb56a0f37b6076e3 P 9e671bcd53514c1ab4dec8f408df65fa: Deleting consensus metadata
I20260809 22:41:17.909634 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:17.879400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.28.243.65:36427)
0809 22:41:17.880218 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:17.880231 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:17.880256 (+ 25us) server_negotiation.cc:407] Connection header received
0809 22:41:17.880331 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:17.880339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:17.880449 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:17.880581 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:17.882673 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.883584 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.894622 (+ 11038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:17.895383 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:17.895544 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:17.899345 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:17.899363 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:17.899383 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:17.899445 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:17.904635 (+ 5190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.905642 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.905653 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.905657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.905849 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:17.906423 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:17.906429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:17.906432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:17.906947 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:17.907315 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:17.908852 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0809 22:41:17.909147 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627}
I20260809 22:41:17.910701 14601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:41:17.911293 14397 catalog_manager.cc:4841] TS 9e671bcd53514c1ab4dec8f408df65fa (127.28.243.65:36427): tablet 2c949f1361354368bb56a0f37b6076e3 (table test-table [id=d33a98e4bb2d41b8a6acc01b42bcd92e]) successfully deleted
I20260809 22:41:18.018313 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 14449
I20260809 22:41:18.041676 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 14612
I20260809 22:41:18.057183 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:18.042533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.28.243.66:39405)
0809 22:41:18.042718 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:18.042722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:18.042742 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:41:18.042824 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:18.042828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:18.042893 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:18.042990 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:18.044374 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:18.045264 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:18.046849 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:18.047303 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:18.047430 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:18.050855 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:18.050870 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:18.050886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:18.050928 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:18.054133 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:18.054915 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:18.054922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:18.054925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:18.055132 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:18.055546 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:18.055549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:18.055551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:18.055948 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:18.056252 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:18.056662 (+ 410us) server_negotiation.cc:299] Negotiation successful
0809 22:41:18.056838 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:41:18.057922 14739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:41:18.153191 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 14612
I20260809 22:41:18.176168 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 14749
I20260809 22:41:18.192230 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:18.177287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.28.243.67:36219)
0809 22:41:18.177491 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:18.177496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:18.177516 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:41:18.177581 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:18.177585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:18.177649 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:18.177738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:18.179338 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:18.180078 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:18.181880 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:18.182353 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:18.182475 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:18.185524 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:18.185540 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:18.185558 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:18.185615 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:18.188813 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:18.189831 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:18.189839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:18.189841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:18.190023 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:18.190516 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:18.190520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:18.190522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:18.190934 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:18.191187 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:18.191531 (+ 344us) server_negotiation.cc:299] Negotiation successful
0809 22:41:18.191713 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:41:18.193084 14879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:41:18.289018 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 14749
I20260809 22:41:18.308169 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 14378
I20260809 22:41:18.309428 14442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:41:18.428032 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 14378
I20260809 22:41:38.454763 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41591
--webserver_interface=127.28.243.126
--webserver_port=38405
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:41591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:41:38.809428 14937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:38.809870 14937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:38.810070 14937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:38.817811 14937 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 22:41:38.817991 14937 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 22:41:38.818091 14937 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 22:41:38.818199 14937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:41:38.818298 14937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:38.818449 14937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:38.818548 14937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:41:38.818639 14937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:41:38.829523 14937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:41591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:41591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:38.831183 14937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:38.832805 14937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:38.846369 14943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:38.846477 14947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:38.846498 14944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:38.847399 14937 server_base.cc:1034] running on GCE node
I20260809 22:41:38.848232 14937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:38.849949 14937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:38.851253 14937 hybrid_clock.cc:648] HybridClock initialized: now 1786315298851172 us; error 55 us; skew 500 ppm
I20260809 22:41:38.851631 14937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:38 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315298, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:38.855137 14937 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:41:38.856844 14937 webserver.cc:460] Webserver started at http://127.28.243.126:38405/ using document root <none> and password file <none>
I20260809 22:41:38.857573 14937 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:38.857666 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:38.863570 14937 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260809 22:41:38.866999 14954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:38.868790 14937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 22:41:38.869020 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "da55f343450447a6ae58c20481c28247"
format_stamp: "Formatted at 2026-08-09 22:40:52 on dist-test-slave-rgb1"
server_key: "d1defc97befb4da55e5642441cfb3c1e"
server_key_iv: "999830aa9bd04b40c50b778db8efa1ba"
server_key_version: "encryptionkey@0"
I20260809 22:41:38.870658 14937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:38.892679 14937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:38.897756 14937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:38.898093 14937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:38.918596 14937 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:41591
I20260809 22:41:38.918689 15008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:41591 every 8 connection(s)
I20260809 22:41:38.920234 14937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:41:38.927623 15009 sys_catalog.cc:263] Verifying existing consensus state
I20260809 22:41:38.928277 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 14937
I20260809 22:41:38.929476 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:36427
--local_ip_for_outbound_sockets=127.28.243.65
--tserver_master_addrs=127.28.243.126:41591
--webserver_port=35107
--webserver_interface=127.28.243.65
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:41:38.933326 15009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Bootstrap starting.
I20260809 22:41:38.957878 15009 log.cc:824] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Log is configured to *not* fsync() on all Append() calls
I20260809 22:41:38.975708 15009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 22:41:38.976353 15009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Bootstrap complete.
I20260809 22:41:38.986131 15009 raft_consensus.cc:348] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:38.987124 15009 raft_consensus.cc:729] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: da55f343450447a6ae58c20481c28247, State: Initialized, Role: FOLLOWER
I20260809 22:41:38.987805 15009 consensus_queue.cc:260] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:38.988077 15009 raft_consensus.cc:388] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:41:38.988209 15009 raft_consensus.cc:482] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:41:38.988413 15009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 2 FOLLOWER]: Advancing to term 3
I20260809 22:41:38.992619 15009 raft_consensus.cc:504] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:38.993186 15009 leader_election.cc:302] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da55f343450447a6ae58c20481c28247; no voters:
I20260809 22:41:38.993954 15009 leader_election.cc:288] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 22:41:38.994089 15013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 22:41:38.995527 15013 raft_consensus.cc:686] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [term 3 LEADER]: Becoming Leader. State: Replica: da55f343450447a6ae58c20481c28247, State: Running, Role: LEADER
I20260809 22:41:38.996125 15013 consensus_queue.cc:237] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } }
I20260809 22:41:38.996430 15009 sys_catalog.cc:564] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:41:38.999094 15015 sys_catalog.cc:455] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da55f343450447a6ae58c20481c28247. Latest consensus state: current_term: 3 leader_uuid: "da55f343450447a6ae58c20481c28247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } } }
I20260809 22:41:38.999208 15014 sys_catalog.cc:455] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "da55f343450447a6ae58c20481c28247" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da55f343450447a6ae58c20481c28247" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 41591 } } }
I20260809 22:41:38.999596 15015 sys_catalog.cc:458] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: This master's current role is: LEADER
I20260809 22:41:38.999680 15014 sys_catalog.cc:458] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247 [sys.catalog]: This master's current role is: LEADER
I20260809 22:41:39.009795 15020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:41:39.021719 15020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:41:39.024487 15020 catalog_manager.cc:1259] Loaded cluster ID: 6da536f7146f4095a7fb3757c1ce8383
I20260809 22:41:39.024703 15020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:41:39.032738 15020 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:41:39.036037 15020 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Loaded TSK: 0
I20260809 22:41:39.036260 15020 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da55f343450447a6ae58c20481c28247: Loaded TSK: 1
F20260809 22:41:39.038053 15020 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f25d2aa69ad google::LogMessage::Fail() at ??:0
@ 0x7f25d2aa988a google::LogMessage::SendToLog() at ??:0
@ 0x7f25d2aa64de google::LogMessage::Flush() at ??:0
@ 0x7f25d2aa7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f25d680a7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f25d67f973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f25e84d51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f25e84d4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f25e85602a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f25e85600c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f25e85aa5e9 std::function<>::operator()() at ??:0
@ 0x7f25e84d8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f25e84d6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f25e855e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f25e855e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f25d56f2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f25d56f591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f25d56f5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f25d56d0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f25d89aa6db start_thread at ??:0
@ 0x7f25d194061f clone at ??:0
W20260809 22:41:39.352332 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:39.352735 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:39.352917 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:39.361294 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:39.361471 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:39.361653 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:41:39.372123 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:36427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=35107
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:39.373646 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:39.375070 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:39.389659 15060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:39.389765 15059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:39.390409 15062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:39.390544 15011 server_base.cc:1034] running on GCE node
I20260809 22:41:39.391335 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:39.392418 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:39.393687 15011 hybrid_clock.cc:648] HybridClock initialized: now 1786315299393649 us; error 89 us; skew 500 ppm
I20260809 22:41:39.394083 15011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:39 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:39.398037 15011 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:41:39.401455 15011 webserver.cc:460] Webserver started at http://127.28.243.65:35107/ using document root <none> and password file <none>
I20260809 22:41:39.402505 15011 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:39.402664 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:39.409674 15011 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 22:41:39.412912 15069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:39.414569 15011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:41:39.414726 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "9e671bcd53514c1ab4dec8f408df65fa"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "9cf6267c200021af7e409cd6b69081ba"
server_key_iv: "346c616d5f1e52741c41765529a8752c"
server_key_version: "encryptionkey@0"
I20260809 22:41:39.415733 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:39.445014 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:39.450268 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:39.450554 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:39.451753 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:39.453249 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:39.453351 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:39.453433 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:39.453497 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:39.494338 15011 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:36427
I20260809 22:41:39.494588 15187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:36427 every 8 connection(s)
I20260809 22:41:39.496232 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:41:39.502229 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:39.499938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:38355)
0809 22:41:39.500894 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:39.501236 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":345,"threads_started":1}
W20260809 22:41:39.503183 15188 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:41591 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.126:41591: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
I20260809 22:41:39.504171 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:39.503774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:35247)
0809 22:41:39.503965 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:39.504055 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 22:41:39.505252 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:39.504843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:56415)
0809 22:41:39.505022 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:39.505112 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260809 22:41:39.505570 15188 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 22:41:39.506793 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 15011
I20260809 22:41:39.507496 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:39405
--local_ip_for_outbound_sockets=127.28.243.66
--tserver_master_addrs=127.28.243.126:41591
--webserver_port=36933
--webserver_interface=127.28.243.66
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:41:39.864966 15191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:39.865365 15191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:39.865562 15191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:39.874262 15191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:39.874439 15191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:39.874614 15191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:41:39.884858 15191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:39405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=36933
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:39.886260 15191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:39.887624 15191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:39.902200 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:39.902258 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:39.902182 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:39.903026 15191 server_base.cc:1034] running on GCE node
I20260809 22:41:39.903720 15191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:39.904863 15191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:39.906106 15191 hybrid_clock.cc:648] HybridClock initialized: now 1786315299906051 us; error 41 us; skew 500 ppm
I20260809 22:41:39.906477 15191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:39 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315299, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:39.909777 15191 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:41:39.911343 15191 webserver.cc:460] Webserver started at http://127.28.243.66:36933/ using document root <none> and password file <none>
I20260809 22:41:39.912043 15191 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:39.912149 15191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:39.917826 15191 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260809 22:41:39.921113 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:39.922766 15191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 22:41:39.922966 15191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "7a69a84ae98344149d5819e354bd52a8"
format_stamp: "Formatted at 2026-08-09 22:40:53 on dist-test-slave-rgb1"
server_key: "5951b09966d22ebe5e1818980e5396b6"
server_key_iv: "8b6bc703e4eeeadf9738acd4987e2c55"
server_key_version: "encryptionkey@0"
I20260809 22:41:39.924009 15191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:39.944363 15191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:39.949270 15191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:39.949575 15191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:39.950609 15191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:39.952033 15191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:39.952121 15191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:39.952277 15191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:39.952365 15191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:39.989935 15191 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:39405
I20260809 22:41:39.989989 15323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:39405 every 8 connection(s)
I20260809 22:41:39.991460 15191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:41:39.996867 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:39.994880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:45625)
0809 22:41:39.995660 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:39.995905 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":304,"threads_started":1}
W20260809 22:41:39.997664 15324 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:41591 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.126:41591: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
I20260809 22:41:39.998610 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:39.998267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:50997)
0809 22:41:39.998403 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:39.998502 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 22:41:39.999516 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:39.999168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:45697)
0809 22:41:39.999308 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:39.999386 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 22:41:39.999787 15324 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 22:41:40.001577 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 15191
I20260809 22:41:40.002604 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:36219
--local_ip_for_outbound_sockets=127.28.243.67
--tserver_master_addrs=127.28.243.126:41591
--webserver_port=45359
--webserver_interface=127.28.243.67
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:41:40.351212 15328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:40.351498 15328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:40.351593 15328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:40.358963 15328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:40.359086 15328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:40.359201 15328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:41:40.369922 15328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:36219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=45359
--tserver_master_addrs=127.28.243.126:41591
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:40.371224 15328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:40.372526 15328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:40.386718 15334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:40.386885 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:40.386855 15336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:40.387897 15328 server_base.cc:1034] running on GCE node
I20260809 22:41:40.388788 15328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:40.389894 15328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:40.391165 15328 hybrid_clock.cc:648] HybridClock initialized: now 1786315300391120 us; error 37 us; skew 500 ppm
I20260809 22:41:40.391523 15328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:40 dist-test-slave-rgb1 krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315300, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:40.395491 15328 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:41:40.397245 15328 webserver.cc:460] Webserver started at http://127.28.243.67:45359/ using document root <none> and password file <none>
I20260809 22:41:40.398008 15328 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:40.398167 15328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:40.404138 15328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 22:41:40.407428 15344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:40.408903 15328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 22:41:40.409130 15328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "8d6ff93a5d754321bf1785536065d58b"
format_stamp: "Formatted at 2026-08-09 22:40:54 on dist-test-slave-rgb1"
server_key: "b836d5663ae05881eb2e0627d34f4aa6"
server_key_iv: "40e47d7e9ef2e001f7e4a6c044061482"
server_key_version: "encryptionkey@0"
I20260809 22:41:40.410212 15328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:40.425014 15328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:40.429752 15328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:40.430049 15328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:40.431170 15328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:40.432649 15328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:40.432773 15328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:40.432902 15328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:40.432991 15328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:40.470386 15328 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:36219
I20260809 22:41:40.470587 15460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:36219 every 8 connection(s)
I20260809 22:41:40.472270 15328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:41:40.478214 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 15328
I20260809 22:41:40.478224 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:40.476022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:35345)
0809 22:41:40.476843 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:40.477094 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":310,"threads_started":1}
W20260809 22:41:40.478937 15461 heartbeater.cc:644] Failed to heartbeat to 127.28.243.126:41591 (0 consecutive failures): Network error: Failed to ping master at 127.28.243.126:41591: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
I20260809 22:41:40.479882 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:40.479502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:43185)
0809 22:41:40.479695 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:40.479769 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 22:41:40.480957 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:40.480632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:47105)
0809 22:41:40.480774 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:40.480850 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 22:41:40.481231 15461 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 22:41:40.507191 15465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:40.506217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:40095)
0809 22:41:40.506785 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:40.506887 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":297,"threads_started":1}
I20260809 22:41:41.001405 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:41.000450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:40577)
0809 22:41:41.001028 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:41.001150 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":294,"threads_started":1}
I20260809 22:41:41.482791 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:41.481841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:33493)
0809 22:41:41.482396 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:41.482549 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
I20260809 22:41:41.509197 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:41.508291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:37665)
0809 22:41:41.508914 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:41.509005 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":368,"threads_started":1}
I20260809 22:41:42.003623 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:42.002633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:44661)
0809 22:41:42.003305 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:42.003418 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":283,"threads_started":1}
I20260809 22:41:42.485004 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:42.484018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:37337)
0809 22:41:42.484653 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:42.484750 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":307,"threads_started":1}
I20260809 22:41:42.511529 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:42.510580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:40721)
0809 22:41:42.511194 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:42.511289 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":373,"threads_started":1}
I20260809 22:41:43.005786 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:43.004939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:43077)
0809 22:41:43.005490 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:43.005581 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":317,"threads_started":1}
I20260809 22:41:43.486835 15473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:43.486047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:55839)
0809 22:41:43.486554 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:43.486656 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260809 22:41:43.513769 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:43.512728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:48099)
0809 22:41:43.513451 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:43.513554 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":450,"threads_started":1}
I20260809 22:41:44.008276 15476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:44.007293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:46451)
0809 22:41:44.007971 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:44.008074 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":284,"threads_started":1}
I20260809 22:41:44.489149 15477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:44.488170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:53271)
0809 22:41:44.488781 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:44.488874 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":296,"threads_started":1}
I20260809 22:41:44.515872 15478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:44.514977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:34389)
0809 22:41:44.515579 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:44.515679 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":279,"threads_started":1}
I20260809 22:41:45.010401 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:45.009527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:54191)
0809 22:41:45.010125 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:45.010216 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260809 22:41:45.491228 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:45.490354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:48703)
0809 22:41:45.490936 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:45.491042 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":295,"threads_started":1}
I20260809 22:41:45.517895 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:45.516988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:57303)
0809 22:41:45.517599 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:45.517721 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":300,"threads_started":1}
I20260809 22:41:46.012307 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:46.011506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:43441)
0809 22:41:46.012030 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:46.012132 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":283,"threads_started":1}
I20260809 22:41:46.493290 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:46.492415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:54991)
0809 22:41:46.492986 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:46.493108 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":329,"threads_started":1}
I20260809 22:41:46.519992 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:46.519030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:33495)
0809 22:41:46.519666 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:46.519758 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":322,"threads_started":1}
I20260809 22:41:47.014516 15485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:47.013503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:38097)
0809 22:41:47.014080 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:47.014177 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":238,"threads_started":1}
I20260809 22:41:47.495536 15486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:47.494532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:60929)
0809 22:41:47.495195 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:47.495290 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":377,"threads_started":1}
I20260809 22:41:47.522033 15487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:47.521209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:53061)
0809 22:41:47.521748 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:47.521840 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":296,"threads_started":1}
I20260809 22:41:48.016542 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:48.015739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:35143)
0809 22:41:48.016264 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:48.016361 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260809 22:41:48.497648 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:48.496689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:42221)
0809 22:41:48.497280 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:48.497405 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":287,"threads_started":1}
I20260809 22:41:48.524081 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:48.523178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:58087)
0809 22:41:48.523786 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:48.523889 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":268,"threads_started":1}
I20260809 22:41:49.018530 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:49.017654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:54663)
0809 22:41:49.018230 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:49.018317 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":251,"threads_started":1}
I20260809 22:41:49.499601 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:49.498772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:58717)
0809 22:41:49.499320 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:49.499413 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":273,"threads_started":1}
I20260809 22:41:49.526051 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:49.525270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:35321)
0809 22:41:49.525789 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:49.525879 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":294,"threads_started":1}
I20260809 22:41:50.021207 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:50.020132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:52755)
0809 22:41:50.020858 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:50.021016 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":346,"threads_started":1}
I20260809 22:41:50.502067 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:50.501036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:53673)
0809 22:41:50.501690 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:50.501794 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":281,"threads_started":1}
I20260809 22:41:50.528496 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:50.527130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:43447)
0809 22:41:50.528050 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:50.528238 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":395,"threads_started":1}
I20260809 22:41:51.023223 15497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:51.022362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:37063)
0809 22:41:51.022933 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:51.023037 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":280,"threads_started":1}
I20260809 22:41:51.504163 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:51.503235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:52909)
0809 22:41:51.503872 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:51.503975 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":312,"threads_started":1}
I20260809 22:41:51.530649 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:51.529791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:36447)
0809 22:41:51.530369 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:51.530472 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":346,"threads_started":1}
I20260809 22:41:52.025218 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:52.024270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:42015)
0809 22:41:52.024857 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:52.024959 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":396,"threads_started":1}
I20260809 22:41:52.506210 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:52.505364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:35687)
0809 22:41:52.505890 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:52.505984 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":292,"threads_started":1}
I20260809 22:41:52.532884 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:52.531778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:57723)
0809 22:41:52.532490 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:52.532673 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":457,"threads_started":1}
I20260809 22:41:53.027293 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:53.026417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:60513)
0809 22:41:53.026942 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:53.027051 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":255,"threads_started":1}
I20260809 22:41:53.508661 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:53.507485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:41205)
0809 22:41:53.508212 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:53.508333 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":370,"threads_started":1}
I20260809 22:41:53.534883 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:53.534065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:50383)
0809 22:41:53.534584 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:53.534688 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":261,"threads_started":1}
I20260809 22:41:54.029623 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:54.028493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:33143)
0809 22:41:54.029194 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:54.029335 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":339,"threads_started":1}
I20260809 22:41:54.510838 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:54.509959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:33809)
0809 22:41:54.510499 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:54.510609 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":265,"threads_started":1}
I20260809 22:41:54.537232 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:54.536303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:44815)
0809 22:41:54.536844 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:54.537042 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":312,"threads_started":1}
I20260809 22:41:55.032053 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:55.030946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.66:52455)
0809 22:41:55.031450 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:55.031760 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
I20260809 22:41:55.485548 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 15011
I20260809 22:41:55.513422 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:55.512281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.67:45657)
0809 22:41:55.513071 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:55.513195 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":482,"threads_started":1}
I20260809 22:41:55.513796 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:55.486823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.28.243.65:36427)
0809 22:41:55.487477 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:55.487498 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:55.487549 (+ 51us) server_negotiation.cc:407] Connection header received
0809 22:41:55.489396 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:55.489407 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:55.489835 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:55.490159 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:55.491771 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:55.493213 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:55.494573 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:55.495364 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:55.495540 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:55.505524 (+ 9984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:55.505535 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:55.505554 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:55.505603 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:55.509468 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:55.510368 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:55.510377 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:55.510384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:55.510597 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:55.511001 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:55.511005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:55.511008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:55.511822 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:55.512107 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:55.512830 (+ 723us) server_negotiation.cc:299] Negotiation successful
0809 22:41:55.513333 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":318,"threads_started":1}
I20260809 22:41:55.515293 15183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:41:55.604530 15513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:55.603339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:41591 (local address 127.28.243.65:42737)
0809 22:41:55.603877 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:55.604166 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.126:41591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":249,"threads_started":1}
I20260809 22:41:55.604521 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 15011
I20260809 22:41:55.625648 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 15191
I20260809 22:41:55.646119 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:55.626831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.28.243.66:39405)
0809 22:41:55.627586 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:55.627606 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:55.627658 (+ 52us) server_negotiation.cc:407] Connection header received
0809 22:41:55.629573 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:55.629582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:55.630043 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:55.630351 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:55.631773 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:55.633267 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:55.634728 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:55.635460 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:55.635615 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:55.638631 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:55.638639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:55.638657 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:55.638701 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:55.642296 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:55.643226 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:55.643233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:55.643241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:55.643437 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:55.643887 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:55.643891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:55.643893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:55.644630 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:55.644889 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:55.645436 (+ 547us) server_negotiation.cc:299] Negotiation successful
0809 22:41:55.645804 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":411,"threads_started":1}
I20260809 22:41:55.647691 15319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:41:55.734678 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 15191
I20260809 22:41:55.757023 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 15328
I20260809 22:41:55.776957 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:55.758268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.28.243.67:36219)
0809 22:41:55.758937 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:55.758951 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:55.758994 (+ 43us) server_negotiation.cc:407] Connection header received
0809 22:41:55.760817 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:55.760828 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:55.761265 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:55.761642 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:55.763166 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:55.764393 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:55.765743 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:55.766422 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:55.766640 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:55.769807 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:55.769816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:55.769832 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:55.769881 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:55.773190 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:55.773991 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:55.773999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:55.774006 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:55.774191 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:55.774612 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:55.774617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:55.774622 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:55.775275 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:55.775599 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:55.776095 (+ 496us) server_negotiation.cc:299] Negotiation successful
0809 22:41:55.776600 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":332,"threads_started":1}
I20260809 22:41:55.778468 15456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:41:55.865985 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 15328
2026-08-09T22:41:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65479 ms)
[----------] 30 tests from SecurityITest (320817 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:41:55 dist-test-slave-rgb1 krb5kdc[15522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:41:55 dist-test-slave-rgb1 krb5kdc[15522](info): set up 2 sockets
Aug 09 22:41:55 dist-test-slave-rgb1 krb5kdc[15522](info): commencing operation
krb5kdc: starting...
W20260809 22:41:57.089094 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:41:57 dist-test-slave-rgb1 krb5kdc[15522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:41:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:41:57.458412 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36579
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:36579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:41:57.817873 15537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:57.818212 15537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:57.818414 15537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:57.826242 15537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:41:57.826373 15537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:57.826476 15537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:57.826594 15537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:41:57.826689 15537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:41:57.837404 15537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:36579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:57.838828 15537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:57.840298 15537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:57.853406 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:57.853561 15546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:57.853570 15543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:57.855311 15537 server_base.cc:1034] running on GCE node
I20260809 22:41:57.856225 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:57.857836 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:57.859091 15537 hybrid_clock.cc:648] HybridClock initialized: now 1786315317859038 us; error 35 us; skew 500 ppm
I20260809 22:41:57.859571 15537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:57 dist-test-slave-rgb1 krb5kdc[15522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315317, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:57.863260 15537 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:41:57.865156 15537 webserver.cc:460] Webserver started at http://127.28.243.126:33885/ using document root <none> and password file <none>
I20260809 22:41:57.865899 15537 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:57.866060 15537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:57.866413 15537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:41:57.870136 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "2f99c2b02caa4511b48e5b04b2ea3946"
format_stamp: "Formatted at 2026-08-09 22:41:57 on dist-test-slave-rgb1"
server_key: "9f755ce50623e00238f48286e3ac3b35"
server_key_iv: "a91a599273494d20fb845d744b2e80ec"
server_key_version: "encryptionkey@0"
I20260809 22:41:57.871044 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "2f99c2b02caa4511b48e5b04b2ea3946"
format_stamp: "Formatted at 2026-08-09 22:41:57 on dist-test-slave-rgb1"
server_key: "9f755ce50623e00238f48286e3ac3b35"
server_key_iv: "a91a599273494d20fb845d744b2e80ec"
server_key_version: "encryptionkey@0"
I20260809 22:41:57.877671 15537 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 22:41:57.881662 15553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:57.883152 15537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:41:57.883402 15537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "2f99c2b02caa4511b48e5b04b2ea3946"
format_stamp: "Formatted at 2026-08-09 22:41:57 on dist-test-slave-rgb1"
server_key: "9f755ce50623e00238f48286e3ac3b35"
server_key_iv: "a91a599273494d20fb845d744b2e80ec"
server_key_version: "encryptionkey@0"
I20260809 22:41:57.883632 15537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:57.920882 15537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:57.926025 15537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:57.926378 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:57.949427 15537 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:36579
I20260809 22:41:57.949427 15608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:36579 every 8 connection(s)
I20260809 22:41:57.951076 15537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:41:57.955226 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 15537
I20260809 22:41:57.955476 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:41:57.955626 15609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:41:57.956362 29645 external_mini_cluster.cc:1428] Setting key b55f76cf2c09ca2812dea8acc986111f
I20260809 22:41:57.971208 15609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946: Bootstrap starting.
Aug 09 22:41:57 dist-test-slave-rgb1 krb5kdc[15522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:41:57.975590 15609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946: Neither blocks nor log segments found. Creating new log.
I20260809 22:41:57.976814 15609 log.cc:824] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946: Log is configured to *not* fsync() on all Append() calls
I20260809 22:41:57.979949 15609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946: No bootstrap required, opened a new log
I20260809 22:41:57.982843 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:57.958799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.28.243.126:36579)
0809 22:41:57.959679 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:57.959691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:57.959736 (+ 45us) server_negotiation.cc:407] Connection header received
0809 22:41:57.961458 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:57.961482 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:57.961912 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:57.962472 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:57.964184 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:57.965640 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:57.967484 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:57.968388 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:57.968756 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:57.973846 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:57.973858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:57.973874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:57.973924 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:57.977926 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:57.978857 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:57.978867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:57.978874 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:57.979107 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:57.979600 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:57.979607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:57.979611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:57.980241 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:57.980678 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:57.981213 (+ 535us) server_negotiation.cc:299] Negotiation successful
0809 22:41:57.981697 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":378,"threads_started":1}
I20260809 22:41:57.989975 15609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36579 } }
I20260809 22:41:57.990417 15609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:41:57.990525 15609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f99c2b02caa4511b48e5b04b2ea3946, State: Initialized, Role: FOLLOWER
I20260809 22:41:57.991101 15609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36579 } }
I20260809 22:41:57.991421 15609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:41:57.991628 15609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:41:57.991828 15609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:41:57.994297 15609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36579 } }
I20260809 22:41:57.994771 15609 leader_election.cc:302] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f99c2b02caa4511b48e5b04b2ea3946; no voters:
I20260809 22:41:57.995373 15609 leader_election.cc:288] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:41:57.995584 15614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:41:57.996315 15614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [term 1 LEADER]: Becoming Leader. State: Replica: 2f99c2b02caa4511b48e5b04b2ea3946, State: Running, Role: LEADER
I20260809 22:41:57.996892 15614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36579 } }
I20260809 22:41:57.997895 15609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:41:58.000178 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36579 } } }
I20260809 22:41:58.000801 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [sys.catalog]: This master's current role is: LEADER
I20260809 22:41:58.000133 15616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f99c2b02caa4511b48e5b04b2ea3946. Latest consensus state: current_term: 1 leader_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f99c2b02caa4511b48e5b04b2ea3946" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36579 } } }
I20260809 22:41:58.001509 15616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946 [sys.catalog]: This master's current role is: LEADER
I20260809 22:41:58.003439 15622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:41:58.009647 15622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:41:58.020282 15622 catalog_manager.cc:1347] Generated new cluster ID: f3b8690b4f3d46009e5da09d1f65ddef
I20260809 22:41:58.020434 15622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:41:58.041914 15622 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:41:58.043296 15622 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:41:58.051903 15622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946: Generated new TSK 0
I20260809 22:41:58.052923 15622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:41:58.142999 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36579
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:41:58.520856 15637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:58.521256 15637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:58.521448 15637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:58.529220 15637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:58.529390 15637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:58.529585 15637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:41:58.539656 15637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:58.541121 15637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:58.542582 15637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:58.558109 15645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:58.558096 15643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:58.558209 15647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:58.559348 15637 server_base.cc:1034] running on GCE node
I20260809 22:41:58.560143 15637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:58.561326 15637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:58.562568 15637 hybrid_clock.cc:648] HybridClock initialized: now 1786315318562522 us; error 52 us; skew 500 ppm
I20260809 22:41:58.562901 15637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:58 dist-test-slave-rgb1 krb5kdc[15522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315318, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:58.566488 15637 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:41:58.568187 15637 webserver.cc:460] Webserver started at http://127.28.243.65:37059/ using document root <none> and password file <none>
I20260809 22:41:58.569015 15637 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:58.569175 15637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:58.569510 15637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:41:58.572657 15637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "2a75de6650bf44afad6212018522c4b0"
format_stamp: "Formatted at 2026-08-09 22:41:58 on dist-test-slave-rgb1"
server_key: "ba3fd39c66bac34f027b8608d05e5ab1"
server_key_iv: "75098bfcfffe0f162e91fdc1b1f9b8fa"
server_key_version: "encryptionkey@0"
I20260809 22:41:58.573518 15637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "2a75de6650bf44afad6212018522c4b0"
format_stamp: "Formatted at 2026-08-09 22:41:58 on dist-test-slave-rgb1"
server_key: "ba3fd39c66bac34f027b8608d05e5ab1"
server_key_iv: "75098bfcfffe0f162e91fdc1b1f9b8fa"
server_key_version: "encryptionkey@0"
I20260809 22:41:58.579174 15637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 22:41:58.583161 15654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:58.584990 15637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 22:41:58.585223 15637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "2a75de6650bf44afad6212018522c4b0"
format_stamp: "Formatted at 2026-08-09 22:41:58 on dist-test-slave-rgb1"
server_key: "ba3fd39c66bac34f027b8608d05e5ab1"
server_key_iv: "75098bfcfffe0f162e91fdc1b1f9b8fa"
server_key_version: "encryptionkey@0"
I20260809 22:41:58.585489 15637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:58.614495 15637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:58.620172 15637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:58.620673 15637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:58.622030 15637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:58.623780 15637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:58.623922 15637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:58.624112 15637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:58.624214 15637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:58.666791 15637 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:45127
I20260809 22:41:58.666851 15770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:45127 every 8 connection(s)
I20260809 22:41:58.668385 15637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:41:58.670203 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 15637
I20260809 22:41:58.670456 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:41:58.671468 29645 external_mini_cluster.cc:1428] Setting key 9015f9b64c90e9652851ac22fa74709b
Aug 09 22:41:58 dist-test-slave-rgb1 krb5kdc[15522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315318, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:41:58.693503 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:58.671873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:37275 (local address 127.28.243.126:36579)
0809 22:41:58.672471 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:58.672478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:58.673748 (+ 1270us) server_negotiation.cc:407] Connection header received
0809 22:41:58.676094 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:58.676101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:58.676224 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:58.676380 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:58.678291 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:58.679122 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:58.680998 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:58.681470 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:58.681637 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:58.685961 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:58.685966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:58.685970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:58.686007 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:58.689841 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:58.690751 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:58.690759 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:58.690769 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:58.690959 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:58.691465 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:58.691470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:58.691474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:58.691785 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:58.691982 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:58.692824 (+ 842us) server_negotiation.cc:299] Negotiation successful
0809 22:41:58.693103 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":306,"threads_started":1}
I20260809 22:41:58.694598 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:58.672524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36579 (local address 127.28.243.65:37275)
0809 22:41:58.673519 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:58.673606 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:58.675745 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:58.676695 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:58.676705 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:58.677436 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:41:58.678077 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:58.678097 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:58.679303 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:58.679309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:58.680836 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:58.680850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:58.681745 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:58.681756 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:58.682034 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:58.682848 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:41:58.682874 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:41:58.685658 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:41:58.690014 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:58.690024 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:58.690044 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:58.690597 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:58.691084 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:58.691089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:58.691093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:58.691314 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:58.691953 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:41:58.691961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:41:58.692367 (+ 406us) client_negotiation.cc:769] Sending connection context
0809 22:41:58.692688 (+ 321us) client_negotiation.cc:240] Negotiation successful
0809 22:41:58.693294 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":400,"threads_started":1}
I20260809 22:41:58.696688 15771 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36579
I20260809 22:41:58.697046 15771 heartbeater.cc:464] Registering TS with master...
I20260809 22:41:58.697921 15771 heartbeater.cc:511] Master 127.28.243.126:36579 requested a full tablet report, sending...
I20260809 22:41:58.700773 15570 ts_manager.cc:194] Registered new tserver with Master: 2a75de6650bf44afad6212018522c4b0 (127.28.243.65:45127)
I20260809 22:41:58.703519 15570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:37275
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:41:58.750744 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36579
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:41:59.115494 15777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:59.115823 15777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:59.116055 15777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:59.123623 15777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:59.123749 15777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:59.123962 15777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:41:59.134938 15777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:59.136421 15777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:59.137892 15777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:59.151683 15786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:59.151865 15784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:59.152475 15788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:59.153460 15777 server_base.cc:1034] running on GCE node
I20260809 22:41:59.154196 15777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:59.155351 15777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:59.156674 15777 hybrid_clock.cc:648] HybridClock initialized: now 1786315319156572 us; error 86 us; skew 500 ppm
I20260809 22:41:59.157150 15777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:59 dist-test-slave-rgb1 krb5kdc[15522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:59.161069 15777 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:41:59.162822 15777 webserver.cc:460] Webserver started at http://127.28.243.66:43615/ using document root <none> and password file <none>
I20260809 22:41:59.163712 15777 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:59.163864 15777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:59.164258 15777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:41:59.168326 15777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "7833f8f8b12b4a94ba879758bb0ca022"
format_stamp: "Formatted at 2026-08-09 22:41:59 on dist-test-slave-rgb1"
server_key: "bf334eb902f5efe84fd324297fb5d39d"
server_key_iv: "dd41bdd8ad7ee919856d984349c24d90"
server_key_version: "encryptionkey@0"
I20260809 22:41:59.169368 15777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "7833f8f8b12b4a94ba879758bb0ca022"
format_stamp: "Formatted at 2026-08-09 22:41:59 on dist-test-slave-rgb1"
server_key: "bf334eb902f5efe84fd324297fb5d39d"
server_key_iv: "dd41bdd8ad7ee919856d984349c24d90"
server_key_version: "encryptionkey@0"
I20260809 22:41:59.177171 15777 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260809 22:41:59.186074 15795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:59.188023 15777 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260809 22:41:59.188273 15777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "7833f8f8b12b4a94ba879758bb0ca022"
format_stamp: "Formatted at 2026-08-09 22:41:59 on dist-test-slave-rgb1"
server_key: "bf334eb902f5efe84fd324297fb5d39d"
server_key_iv: "dd41bdd8ad7ee919856d984349c24d90"
server_key_version: "encryptionkey@0"
I20260809 22:41:59.188520 15777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:59.214255 15777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:59.218765 15777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:59.219030 15777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:59.219992 15777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:59.221362 15777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:59.221441 15777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:59.221524 15777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:59.221571 15777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:59.259178 15777 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:46445
I20260809 22:41:59.259292 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:46445 every 8 connection(s)
I20260809 22:41:59.260753 15777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:41:59.266660 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 15777
I20260809 22:41:59.266880 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:41:59.267418 29645 external_mini_cluster.cc:1428] Setting key 9519649328dfc5c265f90e03559ff9b7
Aug 09 22:41:59 dist-test-slave-rgb1 krb5kdc[15522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:41:59.288437 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.263987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:38933 (local address 127.28.243.126:36579)
0809 22:41:59.264645 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:59.264652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:59.265625 (+ 973us) server_negotiation.cc:407] Connection header received
0809 22:41:59.268354 (+ 2729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:59.268361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:59.268481 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:59.268685 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:59.270514 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.271296 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.274199 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.274825 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.274939 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.280952 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:59.280959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:59.280964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:59.281012 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:59.284021 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:59.285011 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:59.285017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:59.285024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:59.285195 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:59.285865 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:59.285874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:59.285881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:59.286370 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:59.286698 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:59.287577 (+ 879us) server_negotiation.cc:299] Negotiation successful
0809 22:41:59.287880 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
I20260809 22:41:59.289251 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.264611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36579 (local address 127.28.243.66:38933)
0809 22:41:59.265439 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:59.265485 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:59.268035 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:59.268926 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:59.268934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:59.269514 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:41:59.270278 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.270307 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.271480 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.271488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.273975 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.273996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.275093 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.275103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.275450 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.276943 (+ 1493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:41:59.276985 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:41:59.280608 (+ 3623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:41:59.284231 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:59.284240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:59.284255 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:59.284842 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:59.285339 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:59.285343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:59.285345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:59.285723 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:59.286556 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:41:59.286563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:41:59.287027 (+ 464us) client_negotiation.cc:769] Sending connection context
0809 22:41:59.287323 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 22:41:59.287830 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":354,"threads_started":1}
I20260809 22:41:59.290987 15912 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36579
I20260809 22:41:59.291368 15912 heartbeater.cc:464] Registering TS with master...
I20260809 22:41:59.292443 15912 heartbeater.cc:511] Master 127.28.243.126:36579 requested a full tablet report, sending...
I20260809 22:41:59.294282 15570 ts_manager.cc:194] Registered new tserver with Master: 7833f8f8b12b4a94ba879758bb0ca022 (127.28.243.66:46445)
I20260809 22:41:59.295248 15570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:38933
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:41:59.348922 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36579
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:41:59.707440 15771 heartbeater.cc:503] Master 127.28.243.126:36579 was elected leader, sending a full tablet report...
W20260809 22:41:59.722681 15919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:41:59.722973 15919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:41:59.723070 15919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:41:59.730403 15919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:41:59.730533 15919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:41:59.730654 15919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:41:59.740574 15919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:36579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:41:59.741883 15919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:41:59.743187 15919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:41:59.757085 15925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:59.757241 15928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:41:59.757208 15926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:41:59.758699 15919 server_base.cc:1034] running on GCE node
I20260809 22:41:59.759440 15919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:41:59.760432 15919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:41:59.761698 15919 hybrid_clock.cc:648] HybridClock initialized: now 1786315319761645 us; error 63 us; skew 500 ppm
I20260809 22:41:59.762086 15919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:41:59 dist-test-slave-rgb1 krb5kdc[15522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:41:59.765570 15919 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:41:59.767254 15919 webserver.cc:460] Webserver started at http://127.28.243.67:34273/ using document root <none> and password file <none>
I20260809 22:41:59.768046 15919 fs_manager.cc:359] Metadata directory not provided
I20260809 22:41:59.768214 15919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:41:59.768659 15919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:41:59.772092 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "1ed103ee7a4849bca64f102933e44d39"
format_stamp: "Formatted at 2026-08-09 22:41:59 on dist-test-slave-rgb1"
server_key: "4290b9dea136d76b2487b0bba50664bd"
server_key_iv: "b8a2b2b6d47feee96aae266a7007e1da"
server_key_version: "encryptionkey@0"
I20260809 22:41:59.773067 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "1ed103ee7a4849bca64f102933e44d39"
format_stamp: "Formatted at 2026-08-09 22:41:59 on dist-test-slave-rgb1"
server_key: "4290b9dea136d76b2487b0bba50664bd"
server_key_iv: "b8a2b2b6d47feee96aae266a7007e1da"
server_key_version: "encryptionkey@0"
I20260809 22:41:59.779165 15919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:41:59.783578 15936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:59.785353 15919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 22:41:59.785564 15919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "1ed103ee7a4849bca64f102933e44d39"
format_stamp: "Formatted at 2026-08-09 22:41:59 on dist-test-slave-rgb1"
server_key: "4290b9dea136d76b2487b0bba50664bd"
server_key_iv: "b8a2b2b6d47feee96aae266a7007e1da"
server_key_version: "encryptionkey@0"
I20260809 22:41:59.785796 15919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:41:59.823005 15919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:41:59.827773 15919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:41:59.828109 15919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:41:59.829299 15919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:41:59.830751 15919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:41:59.830874 15919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:59.831041 15919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:41:59.831116 15919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:41:59.869529 15919 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:37209
I20260809 22:41:59.869627 16054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:37209 every 8 connection(s)
I20260809 22:41:59.870894 15919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:41:59.873430 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 15919
I20260809 22:41:59.873701 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:41:59.874275 29645 external_mini_cluster.cc:1428] Setting key 68ba93f48b1cfd410ead9a918f2c4e97
Aug 09 22:41:59 dist-test-slave-rgb1 krb5kdc[15522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315319, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:41:59.897445 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.874906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:34647 (local address 127.28.243.126:36579)
0809 22:41:59.881001 (+ 6095us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:59.881006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:59.881024 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:41:59.881077 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:59.881081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:59.881175 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:59.881326 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:59.883007 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.883836 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.885589 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.886038 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.886208 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.890662 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:59.890672 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:59.890678 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:59.890722 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:59.894146 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:59.895040 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:59.895045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:59.895047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:59.895179 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:59.895649 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:59.895653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:59.895655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:59.895984 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:59.896136 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:59.896978 (+ 842us) server_negotiation.cc:299] Negotiation successful
0809 22:41:59.897169 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5982,"thread_start_us":5844,"threads_started":1}
I20260809 22:41:59.898413 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.874337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:36579 (local address 127.28.243.67:34647)
0809 22:41:59.875205 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:59.875268 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:59.878661 (+ 3393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:59.881606 (+ 2945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:59.881616 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:59.882197 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:41:59.882791 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.882822 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.884026 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.884030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.885418 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.885431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.886299 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.886308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.886586 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.887441 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:41:59.887468 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:41:59.890345 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:41:59.894316 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:59.894323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:59.894338 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:59.894827 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:59.895284 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:41:59.895287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:41:59.895290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:41:59.895509 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:41:59.896118 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:41:59.896125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:41:59.896508 (+ 383us) client_negotiation.cc:769] Sending connection context
0809 22:41:59.896852 (+ 344us) client_negotiation.cc:240] Negotiation successful
0809 22:41:59.897264 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":615,"threads_started":1}
I20260809 22:41:59.900146 16055 heartbeater.cc:347] Connected to a master server at 127.28.243.126:36579
I20260809 22:41:59.900463 16055 heartbeater.cc:464] Registering TS with master...
I20260809 22:41:59.901505 16055 heartbeater.cc:511] Master 127.28.243.126:36579 requested a full tablet report, sending...
I20260809 22:41:59.903760 15569 ts_manager.cc:194] Registered new tserver with Master: 1ed103ee7a4849bca64f102933e44d39 (127.28.243.67:37209)
I20260809 22:41:59.904788 15569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:34647
I20260809 22:41:59.916383 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:41:59.935611 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.921413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.28.243.126:36579)
0809 22:41:59.921688 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:59.921696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:59.921945 (+ 249us) server_negotiation.cc:407] Connection header received
0809 22:41:59.922254 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:59.922258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:59.922329 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:59.922428 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:41:59.923937 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.924785 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.926156 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.926636 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.926765 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.929335 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:41:59.929342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:41:59.929346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:41:59.929385 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:41:59.932687 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:59.933523 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:59.933529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:59.933533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:59.933683 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:41:59.934089 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:41:59.934093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:41:59.934095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:41:59.934471 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:41:59.934679 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:59.935068 (+ 389us) server_negotiation.cc:299] Negotiation successful
0809 22:41:59.935261 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260809 22:41:59.941155 15569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:41:59.949108 15569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:41:59.987438 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.963952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45127 (local address 127.0.0.1:34260)
0809 22:41:59.965387 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:59.965448 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:59.965616 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:59.978548 (+ 12932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:59.978553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:59.978591 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:59.979171 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.979190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.981379 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.981384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.984030 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.984274 (+ 244us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.986324 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.986331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.986560 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.986600 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 22:41:59.986674 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 22:41:59.986914 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"mutex_wait_us":105,"thread_start_us":1058,"threads_started":1}
I20260809 22:41:59.987880 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.964327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.28.243.65:45127)
0809 22:41:59.977910 (+ 13583us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:59.977922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:59.977955 (+ 33us) server_negotiation.cc:407] Connection header received
0809 22:41:59.978063 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:59.978074 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:59.978298 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:59.978585 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:59.979373 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.981188 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.984486 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.986046 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.986871 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.987170 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:59.987259 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 22:41:59.987492 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13398,"thread_start_us":13244,"threads_started":1}
I20260809 22:41:59.987963 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.965169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.28.243.67:37209)
0809 22:41:59.978461 (+ 13292us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:59.978469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:59.978503 (+ 34us) server_negotiation.cc:407] Connection header received
0809 22:41:59.978598 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:59.978605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:59.978824 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:59.979048 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:59.980489 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.982210 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.984524 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.985914 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.986097 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.986380 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:59.986708 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 22:41:59.986909 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13119,"thread_start_us":12929,"threads_started":1}
I20260809 22:41:59.988307 16070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.965801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:46445 (local address 127.0.0.1:51914)
0809 22:41:59.976542 (+ 10741us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:59.976585 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:59.976718 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:59.978450 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:59.978458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:59.978522 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:59.979716 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.979731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.981963 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.981974 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.984304 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.984319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.986589 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.986595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.986767 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.986804 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 22:41:59.986880 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 22:41:59.986982 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10585,"mutex_wait_us":107,"thread_start_us":10860,"threads_started":1}
I20260809 22:41:59.988324 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.964363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:37209 (local address 127.0.0.1:60484)
0809 22:41:59.976007 (+ 11644us) negotiation.cc:107] Waiting for socket to connect
0809 22:41:59.976045 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 22:41:59.976459 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:41:59.979051 (+ 2592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:41:59.979063 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:41:59.979097 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:41:59.980243 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.980258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.982386 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.982392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.984302 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:41:59.984319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.986146 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.986153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:41:59.986421 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.986480 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 22:41:59.986663 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 22:41:59.987752 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11393,"mutex_wait_us":907,"thread_start_us":11854,"threads_started":1}
I20260809 22:41:59.988828 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:41:59.966194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.28.243.66:46445)
0809 22:41:59.977349 (+ 11155us) server_negotiation.cc:206] Beginning negotiation
0809 22:41:59.977363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:41:59.977402 (+ 39us) server_negotiation.cc:407] Connection header received
0809 22:41:59.977545 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:41:59.977558 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:41:59.977889 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:41:59.978172 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:41:59.979941 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.981795 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.984485 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:41:59.986063 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:41:59.986208 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:41:59.986423 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 22:41:59.987395 (+ 972us) server_negotiation.cc:299] Negotiation successful
0809 22:41:59.988238 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10993,"thread_start_us":10889,"threads_started":1}
I20260809 22:41:59.992545 15703 tablet_service.cc:1467] Processing CreateTablet for tablet 083e66c12b3c497fa61b1dbc35f859da (DEFAULT_TABLE table=test-table [id=a6b7cd59ff3b4e738a45455d7388e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:41:59.993394 15985 tablet_service.cc:1467] Processing CreateTablet for tablet 083e66c12b3c497fa61b1dbc35f859da (DEFAULT_TABLE table=test-table [id=a6b7cd59ff3b4e738a45455d7388e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:41:59.993494 15845 tablet_service.cc:1467] Processing CreateTablet for tablet 083e66c12b3c497fa61b1dbc35f859da (DEFAULT_TABLE table=test-table [id=a6b7cd59ff3b4e738a45455d7388e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:41:59.994310 15703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083e66c12b3c497fa61b1dbc35f859da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:41:59.994823 15985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083e66c12b3c497fa61b1dbc35f859da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:41:59.994930 15845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 083e66c12b3c497fa61b1dbc35f859da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:00.011525 16075 tablet_bootstrap.cc:492] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Bootstrap starting.
I20260809 22:42:00.012439 16076 tablet_bootstrap.cc:492] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Bootstrap starting.
I20260809 22:42:00.016994 16076 tablet_bootstrap.cc:654] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:00.017508 16077 tablet_bootstrap.cc:492] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Bootstrap starting.
I20260809 22:42:00.018700 16076 log.cc:824] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:00.019088 16075 tablet_bootstrap.cc:654] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:00.020494 16075 log.cc:824] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:00.022392 16077 tablet_bootstrap.cc:654] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:00.022656 16076 tablet_bootstrap.cc:492] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: No bootstrap required, opened a new log
I20260809 22:42:00.023029 16076 ts_tablet_manager.cc:1397] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 22:42:00.024091 16077 log.cc:824] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:00.025791 16075 tablet_bootstrap.cc:492] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: No bootstrap required, opened a new log
I20260809 22:42:00.026139 16075 ts_tablet_manager.cc:1397] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260809 22:42:00.027673 16077 tablet_bootstrap.cc:492] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: No bootstrap required, opened a new log
I20260809 22:42:00.027971 16077 ts_tablet_manager.cc:1397] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260809 22:42:00.035197 16077 raft_consensus.cc:348] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.035648 16077 raft_consensus.cc:374] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:00.035743 16077 raft_consensus.cc:729] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed103ee7a4849bca64f102933e44d39, State: Initialized, Role: FOLLOWER
I20260809 22:42:00.036087 16076 raft_consensus.cc:348] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.036586 16076 raft_consensus.cc:374] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:00.036415 16077 consensus_queue.cc:260] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.036710 16076 raft_consensus.cc:729] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a75de6650bf44afad6212018522c4b0, State: Initialized, Role: FOLLOWER
I20260809 22:42:00.037475 16076 consensus_queue.cc:260] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.037734 16055 heartbeater.cc:503] Master 127.28.243.126:36579 was elected leader, sending a full tablet report...
I20260809 22:42:00.038585 16077 ts_tablet_manager.cc:1428] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260809 22:42:00.039240 16075 raft_consensus.cc:348] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.039849 16075 raft_consensus.cc:374] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:00.039983 16075 raft_consensus.cc:729] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7833f8f8b12b4a94ba879758bb0ca022, State: Initialized, Role: FOLLOWER
I20260809 22:42:00.040805 16075 consensus_queue.cc:260] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.042140 16076 ts_tablet_manager.cc:1428] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Time spent starting tablet: real 0.019s user 0.008s sys 0.012s
I20260809 22:42:00.043385 15912 heartbeater.cc:503] Master 127.28.243.126:36579 was elected leader, sending a full tablet report...
I20260809 22:42:00.044492 16075 ts_tablet_manager.cc:1428] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
W20260809 22:42:00.122795 16056 tablet.cc:2369] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:00.128041 16082 raft_consensus.cc:482] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:42:00.128418 16082 raft_consensus.cc:504] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.130460 16082 leader_election.cc:288] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ed103ee7a4849bca64f102933e44d39 (127.28.243.67:37209), 7833f8f8b12b4a94ba879758bb0ca022 (127.28.243.66:46445)
I20260809 22:42:00.138216 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.131180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:35281 (local address 127.28.243.67:37209)
0809 22:42:00.131447 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.131456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.132391 (+ 935us) server_negotiation.cc:407] Connection header received
0809 22:42:00.132634 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.132639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.132703 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.132807 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:00.133581 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.134359 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.136340 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.137243 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.137443 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.137527 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.137813 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.137955 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 22:42:00.138701 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.131161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:41239 (local address 127.28.243.66:46445)
0809 22:42:00.131554 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.131563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.132279 (+ 716us) server_negotiation.cc:407] Connection header received
0809 22:42:00.132539 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.132544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.132674 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.132786 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:00.133529 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.134317 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.136673 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.137612 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.137807 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.137899 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.138244 (+ 345us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.138390 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260809 22:42:00.138923 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.131152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:37209 (local address 127.28.243.65:35281)
0809 22:42:00.132264 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:00.132279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:00.132438 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:00.132820 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:00.132824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:00.132848 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:00.133387 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.133397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.134503 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.134508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.136157 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.136172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.137474 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.137480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.137640 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.137665 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 22:42:00.137775 (+ 110us) client_negotiation.cc:240] Negotiation successful
0809 22:42:00.137926 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":162,"thread_start_us":502,"threads_started":1}
I20260809 22:42:00.138970 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.131123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:46445 (local address 127.28.243.65:41239)
0809 22:42:00.132133 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:00.132156 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:00.132367 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:00.132804 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:00.132808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:00.132837 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:00.133341 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.133353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.134466 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.134476 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.136354 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.136369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.137869 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.137876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.138058 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.138088 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 22:42:00.138213 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 22:42:00.138363 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":150,"thread_start_us":525,"threads_started":1}
I20260809 22:42:00.139676 16083 raft_consensus.cc:482] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:42:00.140079 16083 raft_consensus.cc:504] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.140242 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083e66c12b3c497fa61b1dbc35f859da" candidate_uuid: "2a75de6650bf44afad6212018522c4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed103ee7a4849bca64f102933e44d39" is_pre_election: true
I20260809 22:42:00.140404 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083e66c12b3c497fa61b1dbc35f859da" candidate_uuid: "2a75de6650bf44afad6212018522c4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7833f8f8b12b4a94ba879758bb0ca022" is_pre_election: true
I20260809 22:42:00.140875 16006 raft_consensus.cc:2454] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a75de6650bf44afad6212018522c4b0 in term 0.
I20260809 22:42:00.140936 15866 raft_consensus.cc:2454] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a75de6650bf44afad6212018522c4b0 in term 0.
I20260809 22:42:00.141778 15657 leader_election.cc:302] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed103ee7a4849bca64f102933e44d39, 2a75de6650bf44afad6212018522c4b0; no voters:
I20260809 22:42:00.142271 16082 raft_consensus.cc:2781] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:42:00.142445 16082 raft_consensus.cc:482] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:42:00.142575 16082 raft_consensus.cc:3037] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:00.146878 16082 raft_consensus.cc:504] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.148270 16082 leader_election.cc:288] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [CANDIDATE]: Term 1 election: Requested vote from peers 1ed103ee7a4849bca64f102933e44d39 (127.28.243.67:37209), 7833f8f8b12b4a94ba879758bb0ca022 (127.28.243.66:46445)
I20260809 22:42:00.148389 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083e66c12b3c497fa61b1dbc35f859da" candidate_uuid: "2a75de6650bf44afad6212018522c4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed103ee7a4849bca64f102933e44d39"
I20260809 22:42:00.148742 16006 raft_consensus.cc:3037] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:00.151428 16006 raft_consensus.cc:2454] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a75de6650bf44afad6212018522c4b0 in term 1.
I20260809 22:42:00.151741 16083 leader_election.cc:288] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a75de6650bf44afad6212018522c4b0 (127.28.243.65:45127), 1ed103ee7a4849bca64f102933e44d39 (127.28.243.67:37209)
I20260809 22:42:00.152009 15657 leader_election.cc:302] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed103ee7a4849bca64f102933e44d39, 2a75de6650bf44afad6212018522c4b0; no voters:
I20260809 22:42:00.152400 16082 raft_consensus.cc:2781] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:00.152585 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083e66c12b3c497fa61b1dbc35f859da" candidate_uuid: "2a75de6650bf44afad6212018522c4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7833f8f8b12b4a94ba879758bb0ca022"
I20260809 22:42:00.152912 15866 raft_consensus.cc:3037] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:00.155993 16082 raft_consensus.cc:686] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a75de6650bf44afad6212018522c4b0, State: Running, Role: LEADER
I20260809 22:42:00.156515 16082 consensus_queue.cc:237] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } }
I20260809 22:42:00.161679 15866 raft_consensus.cc:2454] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a75de6650bf44afad6212018522c4b0 in term 1.
I20260809 22:42:00.163082 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.143158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:59831 (local address 127.28.243.65:45127)
0809 22:42:00.143420 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.143428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.143518 (+ 90us) server_negotiation.cc:407] Connection header received
0809 22:42:00.143853 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.143862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.144077 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.144167 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:00.145319 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.146516 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.159921 (+ 13405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.161528 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.161689 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.161784 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.162592 (+ 808us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.162761 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:42:00.164036 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.142590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45127 (local address 127.28.243.66:59831)
0809 22:42:00.143406 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:00.143428 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:00.143657 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:00.144399 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:00.144404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:00.144433 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:00.144886 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.144899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.146760 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.146769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.159676 (+ 12907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.159693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.162111 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.162118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.162355 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.162406 (+ 51us) client_negotiation.cc:769] Sending connection context
0809 22:42:00.163444 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0809 22:42:00.163629 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":844,"threads_started":1}
I20260809 22:42:00.165307 15724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083e66c12b3c497fa61b1dbc35f859da" candidate_uuid: "7833f8f8b12b4a94ba879758bb0ca022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a75de6650bf44afad6212018522c4b0" is_pre_election: true
I20260809 22:42:00.166596 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.142620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:37003 (local address 127.28.243.67:37209)
0809 22:42:00.142829 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.142838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.157694 (+ 14856us) server_negotiation.cc:407] Connection header received
0809 22:42:00.157966 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.157972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.158061 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.158165 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:00.159327 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.160749 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.163722 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.165201 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.165377 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.165462 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.166156 (+ 694us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.166315 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:42:00.166721 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.142716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:37209 (local address 127.28.243.66:37003)
0809 22:42:00.157503 (+ 14787us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:00.157544 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:00.157777 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:00.158345 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:00.158351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:00.158380 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:00.159086 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.159102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.161113 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.161119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.163508 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:00.163524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.165710 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.165718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:00.165939 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.165977 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 22:42:00.166109 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 22:42:00.166274 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14393,"thread_start_us":14151,"threads_started":1}
I20260809 22:42:00.166287 15568 catalog_manager.cc:5495] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a75de6650bf44afad6212018522c4b0 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "2a75de6650bf44afad6212018522c4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a75de6650bf44afad6212018522c4b0" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 } health_report { overall_health: UNKNOWN } } }
I20260809 22:42:00.167476 16006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "083e66c12b3c497fa61b1dbc35f859da" candidate_uuid: "7833f8f8b12b4a94ba879758bb0ca022" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed103ee7a4849bca64f102933e44d39" is_pre_election: true
I20260809 22:42:00.167820 16006 raft_consensus.cc:2379] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7833f8f8b12b4a94ba879758bb0ca022 in current term 1: Already voted for candidate 2a75de6650bf44afad6212018522c4b0 in this term.
I20260809 22:42:00.168895 15798 leader_election.cc:302] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7833f8f8b12b4a94ba879758bb0ca022; no voters: 1ed103ee7a4849bca64f102933e44d39, 2a75de6650bf44afad6212018522c4b0
I20260809 22:42:00.169411 16083 raft_consensus.cc:2726] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 22:42:00.171483 15772 tablet.cc:2369] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:00.200183 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.193296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.28.243.65:45127)
0809 22:42:00.193546 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.193554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.193718 (+ 164us) server_negotiation.cc:407] Connection header received
0809 22:42:00.194069 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.194075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.194185 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.194331 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:42:00.194755 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.195686 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.197439 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.198051 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.198179 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.198772 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:42:00.199395 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:42:00.199540 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.199673 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.199754 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260809 22:42:00.209442 16006 raft_consensus.cc:1261] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 1 FOLLOWER]: Refusing update from remote peer 2a75de6650bf44afad6212018522c4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:00.209709 15866 raft_consensus.cc:1261] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 1 FOLLOWER]: Refusing update from remote peer 2a75de6650bf44afad6212018522c4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:00.210443 16088 consensus_queue.cc:1035] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed103ee7a4849bca64f102933e44d39" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:00.210866 16082 consensus_queue.cc:1035] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7833f8f8b12b4a94ba879758bb0ca022" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 46445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:00.223500 16091 mvcc.cc:205] Tried to move back new op lower bound from 7316747551567106048 to 7316747551366402048. Current Snapshot: MvccSnapshot[applied={T|T < 7316747551567106048}]
I20260809 22:42:00.226001 16092 mvcc.cc:205] Tried to move back new op lower bound from 7316747551567106048 to 7316747551366402048. Current Snapshot: MvccSnapshot[applied={T|T < 7316747551567106048}]
I20260809 22:42:00.240195 15570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:00.240522 15570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:00.244940 15570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f99c2b02caa4511b48e5b04b2ea3946: Sending DeleteTablet for 3 replicas of tablet 083e66c12b3c497fa61b1dbc35f859da
I20260809 22:42:00.246592 15985 tablet_service.cc:1514] Processing DeleteTablet for tablet 083e66c12b3c497fa61b1dbc35f859da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:00 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:60484
I20260809 22:42:00.246727 15845 tablet_service.cc:1514] Processing DeleteTablet for tablet 083e66c12b3c497fa61b1dbc35f859da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:00 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:51914
I20260809 22:42:00.246888 15703 tablet_service.cc:1514] Processing DeleteTablet for tablet 083e66c12b3c497fa61b1dbc35f859da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:00 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:34260
I20260809 22:42:00.249032 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 15637
I20260809 22:42:00.257899 16104 tablet_replica.cc:330] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: stopping tablet replica
I20260809 22:42:00.259043 16103 tablet_replica.cc:330] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: stopping tablet replica
I20260809 22:42:00.259866 16104 raft_consensus.cc:2229] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:42:00.261171 16103 raft_consensus.cc:2229] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:00.261863 16104 raft_consensus.cc:2258] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:00.262727 16103 raft_consensus.cc:2258] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:00.264123 16102 tablet_replica.cc:330] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: stopping tablet replica
I20260809 22:42:00.264638 16102 raft_consensus.cc:2229] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:00.265242 16102 raft_consensus.cc:2258] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:00.266093 16103 ts_tablet_manager.cc:1905] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:00.274729 16102 ts_tablet_manager.cc:1905] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 22:42:00 dist-test-slave-rgb1 krb5kdc[15522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:42:00.278251 16104 ts_tablet_manager.cc:1905] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:00.282702 16103 ts_tablet_manager.cc:1918] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:00.282925 16103 log.cc:1196] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/083e66c12b3c497fa61b1dbc35f859da
I20260809 22:42:00.283597 16103 ts_tablet_manager.cc:1939] T 083e66c12b3c497fa61b1dbc35f859da P 7833f8f8b12b4a94ba879758bb0ca022: Deleting consensus metadata
I20260809 22:42:00.286202 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.250641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.28.243.65:45127)
0809 22:42:00.250872 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.250879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.264264 (+ 13385us) server_negotiation.cc:407] Connection header received
0809 22:42:00.264819 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.264841 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.264944 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.265021 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:00.267001 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.268125 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.270193 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.271030 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.271164 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.277223 (+ 6059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:00.277236 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:00.277256 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:00.277305 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:00.282028 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:00.283217 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:00.283231 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:00.283237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:00.283421 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:00.283958 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:00.283963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:00.283967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:00.284525 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:00.284906 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.285560 (+ 654us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.285810 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 22:42:00.286646 16104 ts_tablet_manager.cc:1918] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:00.286826 16104 log.cc:1196] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/083e66c12b3c497fa61b1dbc35f859da
I20260809 22:42:00.287474 16104 ts_tablet_manager.cc:1939] T 083e66c12b3c497fa61b1dbc35f859da P 2a75de6650bf44afad6212018522c4b0: Deleting consensus metadata
I20260809 22:42:00.287845 16102 ts_tablet_manager.cc:1918] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:00.288085 15766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:00.288057 16102 log.cc:1196] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/083e66c12b3c497fa61b1dbc35f859da
I20260809 22:42:00.289216 16102 ts_tablet_manager.cc:1939] T 083e66c12b3c497fa61b1dbc35f859da P 1ed103ee7a4849bca64f102933e44d39: Deleting consensus metadata
I20260809 22:42:00.291386 15554 catalog_manager.cc:4841] TS 7833f8f8b12b4a94ba879758bb0ca022 (127.28.243.66:46445): tablet 083e66c12b3c497fa61b1dbc35f859da (table test-table [id=a6b7cd59ff3b4e738a45455d7388e022]) successfully deleted
I20260809 22:42:00.292272 15556 catalog_manager.cc:4841] TS 1ed103ee7a4849bca64f102933e44d39 (127.28.243.67:37209): tablet 083e66c12b3c497fa61b1dbc35f859da (table test-table [id=a6b7cd59ff3b4e738a45455d7388e022]) successfully deleted
I20260809 22:42:00.402376 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 15637
I20260809 22:42:00.402468 15554 catalog_manager.cc:4841] TS 2a75de6650bf44afad6212018522c4b0 (127.28.243.65:45127): tablet 083e66c12b3c497fa61b1dbc35f859da (table test-table [id=a6b7cd59ff3b4e738a45455d7388e022]) successfully deleted
I20260809 22:42:00.426142 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 15777
Aug 09 22:42:00 dist-test-slave-rgb1 krb5kdc[15522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:42:00.445087 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.427245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.28.243.66:46445)
0809 22:42:00.427497 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.427503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.427525 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:42:00.427704 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.427709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.427787 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.427898 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:00.429616 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.430397 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.431831 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.432241 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.432434 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.437916 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:00.437935 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:00.437952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:00.438005 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:00.441734 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:00.442638 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:00.442647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:00.442650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:00.442809 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:00.443236 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:00.443243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:00.443246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:00.443681 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:00.444047 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.444383 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.444675 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 22:42:00.445817 15907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:00.544111 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 15777
I20260809 22:42:00.564674 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 15919
Aug 09 22:42:00 dist-test-slave-rgb1 krb5kdc[15522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:42:00.591473 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:00.565606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.28.243.67:37209)
0809 22:42:00.565856 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:00.565865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:00.566045 (+ 180us) server_negotiation.cc:407] Connection header received
0809 22:42:00.566352 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:00.566358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:00.566451 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:00.566591 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:00.568303 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.569240 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.570827 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:00.571351 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:00.571523 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:00.582255 (+ 10732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:00.582274 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:00.582292 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:00.582346 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:00.586407 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:00.587819 (+ 1412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:00.587831 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:00.587834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:00.588057 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:00.588765 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:00.588771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:00.588774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:00.589465 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:00.590195 (+ 730us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:00.590730 (+ 535us) server_negotiation.cc:299] Negotiation successful
0809 22:42:00.591066 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 22:42:00.592270 16050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:00.686535 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 15919
I20260809 22:42:00.707672 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 15537
I20260809 22:42:00.708969 15604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:00.826831 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 15537
2026-08-09T22:42:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4973 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T22:42:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:00Z Disabled control of system clock
I20260809 22:42:00.933362 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38511
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:38511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 22:42:01.311239 16113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:01.311563 16113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:01.311657 16113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:01.319232 16113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:01.319372 16113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:01.319412 16113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:01.319466 16113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:01.319504 16113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:01.330169 16113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:38511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:01.331522 16113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:01.333423 16113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:01.347086 16118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:01.347019 16119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:01.347033 16122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:01.348414 16113 server_base.cc:1034] running on GCE node
I20260809 22:42:01.349500 16113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:01.351212 16113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:01.352494 16113 hybrid_clock.cc:648] HybridClock initialized: now 1786315321352425 us; error 62 us; skew 500 ppm
I20260809 22:42:01.353305 16113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:01.356307 16113 webserver.cc:460] Webserver started at http://127.28.243.126:45591/ using document root <none> and password file <none>
I20260809 22:42:01.357211 16113 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:01.357369 16113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:01.357739 16113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:01.361097 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "551e4255d3104ccdb22d19cd24ee0fe0"
format_stamp: "Formatted at 2026-08-09 22:42:01 on dist-test-slave-rgb1"
server_key: "d685da197329b2967d1b82dc534fa05a"
server_key_iv: "625ce24115cc8ba1efd7f583dd45ab72"
server_key_version: "encryptionkey@0"
I20260809 22:42:01.362056 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "551e4255d3104ccdb22d19cd24ee0fe0"
format_stamp: "Formatted at 2026-08-09 22:42:01 on dist-test-slave-rgb1"
server_key: "d685da197329b2967d1b82dc534fa05a"
server_key_iv: "625ce24115cc8ba1efd7f583dd45ab72"
server_key_version: "encryptionkey@0"
I20260809 22:42:01.368052 16113 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260809 22:42:01.372452 16128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:01.374123 16113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:01.374370 16113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "551e4255d3104ccdb22d19cd24ee0fe0"
format_stamp: "Formatted at 2026-08-09 22:42:01 on dist-test-slave-rgb1"
server_key: "d685da197329b2967d1b82dc534fa05a"
server_key_iv: "625ce24115cc8ba1efd7f583dd45ab72"
server_key_version: "encryptionkey@0"
I20260809 22:42:01.374651 16113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:01.413823 16113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:01.414892 16113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:01.415230 16113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:01.436934 16113 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:38511
I20260809 22:42:01.437038 16182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:38511 every 8 connection(s)
I20260809 22:42:01.438380 16113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:01.439929 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 16113
I20260809 22:42:01.440155 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:01.440814 29645 external_mini_cluster.cc:1428] Setting key fcaff033590398bc5731a8f679658a70
I20260809 22:42:01.443787 16183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:01.456311 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:01.443139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.28.243.126:38511)
0809 22:42:01.444342 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:01.444353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:01.444396 (+ 43us) server_negotiation.cc:407] Connection header received
0809 22:42:01.446224 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:01.446247 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:01.446725 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:01.447167 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:01.447767 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:01.449226 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:01.451302 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:01.452029 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:01.452289 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:01.453703 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:01.453711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:01.453727 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:01.454428 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:01.454519 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:01.454550 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:01.454710 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:01.454956 (+ 246us) server_negotiation.cc:299] Negotiation successful
0809 22:42:01.455310 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":442,"threads_started":1}
I20260809 22:42:01.457170 16183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0: Bootstrap starting.
I20260809 22:42:01.460654 16183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:01.461931 16183 log.cc:824] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:01.465461 16183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0: No bootstrap required, opened a new log
I20260809 22:42:01.474786 16183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38511 } }
I20260809 22:42:01.475217 16183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:01.475304 16183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551e4255d3104ccdb22d19cd24ee0fe0, State: Initialized, Role: FOLLOWER
I20260809 22:42:01.475871 16183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38511 } }
I20260809 22:42:01.476092 16183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:01.476204 16183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:01.476363 16183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:01.478993 16183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38511 } }
I20260809 22:42:01.479579 16183 leader_election.cc:302] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 551e4255d3104ccdb22d19cd24ee0fe0; no voters:
I20260809 22:42:01.480376 16183 leader_election.cc:288] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:01.480497 16188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:01.481572 16188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 551e4255d3104ccdb22d19cd24ee0fe0, State: Running, Role: LEADER
I20260809 22:42:01.482306 16188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38511 } }
I20260809 22:42:01.483415 16183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:01.486016 16189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38511 } } }
I20260809 22:42:01.486480 16189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:01.485952 16190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 551e4255d3104ccdb22d19cd24ee0fe0. Latest consensus state: current_term: 1 leader_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "551e4255d3104ccdb22d19cd24ee0fe0" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38511 } } }
I20260809 22:42:01.486894 16190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:01.489455 16196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:01.494792 16196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:01.504252 16196 catalog_manager.cc:1347] Generated new cluster ID: 74cee94af2c349a38ccd092e0b23abfb
I20260809 22:42:01.504408 16196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:01.520118 16196 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:01.521574 16196 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:01.534978 16196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0: Generated new TSK 0
I20260809 22:42:01.536198 16196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:01.557036 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38511
--builtin_ntp_servers=127.28.243.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 22:42:01.938247 16209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:01.938572 16209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:01.938663 16209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:01.946105 16209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:01.946225 16209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:01.946339 16209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:42:01.955776 16209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:01.957275 16209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:01.960788 16209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:01.975736 16215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:01.975670 16218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:01.975694 16216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:01.976895 16209 server_base.cc:1034] running on GCE node
I20260809 22:42:01.977569 16209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:01.978632 16209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:01.979974 16209 hybrid_clock.cc:648] HybridClock initialized: now 1786315321979919 us; error 33 us; skew 500 ppm
I20260809 22:42:01.980412 16209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:01.983379 16209 webserver.cc:460] Webserver started at http://127.28.243.65:44999/ using document root <none> and password file <none>
I20260809 22:42:01.984210 16209 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:01.984359 16209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:01.984844 16209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:01.988152 16209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "8a5ccaca61624917a5ca1ac7f495da4e"
format_stamp: "Formatted at 2026-08-09 22:42:01 on dist-test-slave-rgb1"
server_key: "cd2cd3cd1aa2d3e2294744f6feef0165"
server_key_iv: "ced83382fef4a029f02949e73fdbd5e9"
server_key_version: "encryptionkey@0"
I20260809 22:42:01.989228 16209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "8a5ccaca61624917a5ca1ac7f495da4e"
format_stamp: "Formatted at 2026-08-09 22:42:01 on dist-test-slave-rgb1"
server_key: "cd2cd3cd1aa2d3e2294744f6feef0165"
server_key_iv: "ced83382fef4a029f02949e73fdbd5e9"
server_key_version: "encryptionkey@0"
I20260809 22:42:01.995558 16209 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260809 22:42:01.999702 16224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:02.001451 16209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:02.001669 16209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "8a5ccaca61624917a5ca1ac7f495da4e"
format_stamp: "Formatted at 2026-08-09 22:42:01 on dist-test-slave-rgb1"
server_key: "cd2cd3cd1aa2d3e2294744f6feef0165"
server_key_iv: "ced83382fef4a029f02949e73fdbd5e9"
server_key_version: "encryptionkey@0"
I20260809 22:42:02.001924 16209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:02.021399 16209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:02.022451 16209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:02.022882 16209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:02.024029 16209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:02.026219 16209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:02.026377 16209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:02.026531 16209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:02.026633 16209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:02.066260 16209 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:45257
I20260809 22:42:02.066308 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:45257 every 8 connection(s)
I20260809 22:42:02.067658 16209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:42:02.074199 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 16209
I20260809 22:42:02.074414 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:42:02.075065 29645 external_mini_cluster.cc:1428] Setting key e706f9e73088f9c8036d6edcd4c52b4f
I20260809 22:42:02.078634 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38511
--builtin_ntp_servers=127.28.243.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 22:42:02.083353 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:02.070489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:48845 (local address 127.28.243.126:38511)
0809 22:42:02.071024 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:02.071031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:02.072179 (+ 1148us) server_negotiation.cc:407] Connection header received
0809 22:42:02.074393 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:02.074400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:02.074526 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:02.074643 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:02.076070 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.077135 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.079703 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.080224 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.080348 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:02.082334 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:02.082340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:02.082344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:02.082365 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:02.082419 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:02.082433 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:02.082546 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:02.082813 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 22:42:02.082929 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":247,"threads_started":1}
I20260809 22:42:02.085150 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:02.071005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38511 (local address 127.28.243.65:48845)
0809 22:42:02.072008 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:02.072052 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:02.073988 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:02.074960 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:02.074972 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:02.075112 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:02.075885 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:02.075905 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.077303 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.077307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:02.079545 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:02.079560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.080486 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.080496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:02.080896 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:02.081913 (+ 1017us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:02.081917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:02.081971 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:02.082074 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:02.082546 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:02.082563 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:02.082575 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 22:42:02.082800 (+ 225us) client_negotiation.cc:240] Negotiation successful
0809 22:42:02.083305 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":332,"threads_started":1}
I20260809 22:42:02.086948 16346 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38511
I20260809 22:42:02.087316 16346 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:02.088212 16346 heartbeater.cc:511] Master 127.28.243.126:38511 requested a full tablet report, sending...
I20260809 22:42:02.091138 16146 ts_manager.cc:194] Registered new tserver with Master: 8a5ccaca61624917a5ca1ac7f495da4e (127.28.243.65:45257)
I20260809 22:42:02.093788 16146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.65:48845
W20260809 22:42:02.449263 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:02.449613 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:02.449807 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:02.458179 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:02.458349 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:02.458557 16350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:42:02.469300 16350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:02.470927 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:02.473007 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:02.487583 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:02.487510 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:02.488018 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:02.488550 16350 server_base.cc:1034] running on GCE node
I20260809 22:42:02.489369 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:02.490568 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:02.491793 16350 hybrid_clock.cc:648] HybridClock initialized: now 1786315322491705 us; error 88 us; skew 500 ppm
I20260809 22:42:02.492204 16350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:02.495133 16350 webserver.cc:460] Webserver started at http://127.28.243.66:39551/ using document root <none> and password file <none>
I20260809 22:42:02.495886 16350 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:02.495983 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:02.496376 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:02.499598 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "4d32d14c8670430a9c959df5aa4baeee"
format_stamp: "Formatted at 2026-08-09 22:42:02 on dist-test-slave-rgb1"
server_key: "493c3e24db9ee4779015e3d884a43517"
server_key_iv: "6a194d3030a48e594fa59cf274c4714c"
server_key_version: "encryptionkey@0"
I20260809 22:42:02.500443 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "4d32d14c8670430a9c959df5aa4baeee"
format_stamp: "Formatted at 2026-08-09 22:42:02 on dist-test-slave-rgb1"
server_key: "493c3e24db9ee4779015e3d884a43517"
server_key_iv: "6a194d3030a48e594fa59cf274c4714c"
server_key_version: "encryptionkey@0"
I20260809 22:42:02.506119 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 22:42:02.510354 16364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:02.511953 16350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:42:02.512279 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "4d32d14c8670430a9c959df5aa4baeee"
format_stamp: "Formatted at 2026-08-09 22:42:02 on dist-test-slave-rgb1"
server_key: "493c3e24db9ee4779015e3d884a43517"
server_key_iv: "6a194d3030a48e594fa59cf274c4714c"
server_key_version: "encryptionkey@0"
I20260809 22:42:02.512558 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:02.526281 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:02.527242 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:02.527601 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:02.528775 16350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:02.530400 16350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:02.530542 16350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:02.530697 16350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:02.530799 16350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:02.571488 16350 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:34585
I20260809 22:42:02.571543 16481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:34585 every 8 connection(s)
I20260809 22:42:02.573168 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:42:02.575657 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 16350
I20260809 22:42:02.575848 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:42:02.576467 29645 external_mini_cluster.cc:1428] Setting key 6316140ef1b4ce5dba3fc9f2ae8e1f3d
I20260809 22:42:02.580068 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38511
--builtin_ntp_servers=127.28.243.84:45185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 22:42:02.590524 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:02.576234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:35857 (local address 127.28.243.126:38511)
0809 22:42:02.576449 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:02.576456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:02.578107 (+ 1651us) server_negotiation.cc:407] Connection header received
0809 22:42:02.580934 (+ 2827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:02.580938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:02.581069 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:02.581170 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:02.582722 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.584151 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.586789 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.587443 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.587580 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:02.589443 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:02.589451 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:02.589458 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:02.589486 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:02.589527 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:02.589544 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:02.589626 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:02.589980 (+ 354us) server_negotiation.cc:299] Negotiation successful
0809 22:42:02.590108 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 22:42:02.591539 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:02.576750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38511 (local address 127.28.243.66:35857)
0809 22:42:02.577887 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:02.577930 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:02.580549 (+ 2619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:02.581455 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:02.581470 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:02.581610 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:02.582490 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:02.582521 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.584377 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.584386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:02.586623 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:02.586643 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:02.587705 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:02.587715 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:02.588079 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:02.589034 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:02.589040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:02.589108 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:02.589227 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:02.589682 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:02.589695 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:02.589706 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 22:42:02.589890 (+ 184us) client_negotiation.cc:240] Negotiation successful
0809 22:42:02.590403 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":411,"threads_started":1}
I20260809 22:42:02.593111 16482 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38511
I20260809 22:42:02.593474 16482 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:02.594254 16482 heartbeater.cc:511] Master 127.28.243.126:38511 requested a full tablet report, sending...
I20260809 22:42:02.595919 16146 ts_manager.cc:194] Registered new tserver with Master: 4d32d14c8670430a9c959df5aa4baeee (127.28.243.66:34585)
I20260809 22:42:02.596913 16146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.66:35857
W20260809 22:42:02.935835 16486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:02.936156 16486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:02.936319 16486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:02.943840 16486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:02.944012 16486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:02.944201 16486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:42:02.953647 16486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:02.955060 16486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:02.956925 16486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:02.971352 16491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:02.971477 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:02.971422 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:02.978721 16486 server_base.cc:1034] running on GCE node
I20260809 22:42:02.979760 16486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:02.981124 16486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:02.982421 16486 hybrid_clock.cc:648] HybridClock initialized: now 1786315322982382 us; error 53 us; skew 500 ppm
I20260809 22:42:02.982827 16486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:02.985858 16486 webserver.cc:460] Webserver started at http://127.28.243.67:33249/ using document root <none> and password file <none>
I20260809 22:42:02.986666 16486 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:02.986824 16486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:02.987205 16486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:02.990743 16486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "ccc89d988dd64eebb9ecce0f7ed6302f"
format_stamp: "Formatted at 2026-08-09 22:42:02 on dist-test-slave-rgb1"
server_key: "e97d6d5ec064c38d1b2f0624c35c45e7"
server_key_iv: "91b630c84df5aea1ec7424cf22e82a28"
server_key_version: "encryptionkey@0"
I20260809 22:42:02.991631 16486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "ccc89d988dd64eebb9ecce0f7ed6302f"
format_stamp: "Formatted at 2026-08-09 22:42:02 on dist-test-slave-rgb1"
server_key: "e97d6d5ec064c38d1b2f0624c35c45e7"
server_key_iv: "91b630c84df5aea1ec7424cf22e82a28"
server_key_version: "encryptionkey@0"
I20260809 22:42:02.997673 16486 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260809 22:42:03.001533 16501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:03.002959 16486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:42:03.003161 16486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "ccc89d988dd64eebb9ecce0f7ed6302f"
format_stamp: "Formatted at 2026-08-09 22:42:02 on dist-test-slave-rgb1"
server_key: "e97d6d5ec064c38d1b2f0624c35c45e7"
server_key_iv: "91b630c84df5aea1ec7424cf22e82a28"
server_key_version: "encryptionkey@0"
I20260809 22:42:03.003397 16486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:03.022692 16486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:03.023643 16486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:03.023983 16486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:03.025215 16486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:03.026773 16486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:03.026912 16486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:03.027055 16486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:03.027148 16486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:03.063583 16486 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:38383
I20260809 22:42:03.063678 16617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:38383 every 8 connection(s)
I20260809 22:42:03.065006 16486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:42:03.065964 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 16486
I20260809 22:42:03.066161 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:42:03.066771 29645 external_mini_cluster.cc:1428] Setting key c3574774ea4ee9a731052c0ee9766fcd
I20260809 22:42:03.080994 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.070059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:59461 (local address 127.28.243.126:38511)
0809 22:42:03.070230 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.070236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.071376 (+ 1140us) server_negotiation.cc:407] Connection header received
0809 22:42:03.073660 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.073664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.073733 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.073824 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:03.075086 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.075848 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.077806 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.078221 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.078343 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.079977 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:03.079982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:03.079986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:03.080044 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:03.080077 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:03.080091 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:03.080168 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.080496 (+ 328us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.080620 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 22:42:03.082422 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.069247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38511 (local address 127.28.243.67:59461)
0809 22:42:03.071226 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.071276 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:03.073393 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:03.074155 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:03.074164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:03.074252 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:03.074847 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.074865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.076092 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.076096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.077631 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.077646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.078463 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.078471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.078715 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.079606 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:03.079609 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:03.079650 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:03.079730 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:03.080205 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:03.080215 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:03.080221 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 22:42:03.080455 (+ 234us) client_negotiation.cc:240] Negotiation successful
0809 22:42:03.080873 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":1108,"threads_started":1}
I20260809 22:42:03.083796 16618 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38511
I20260809 22:42:03.084141 16618 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:03.085057 16618 heartbeater.cc:511] Master 127.28.243.126:38511 requested a full tablet report, sending...
I20260809 22:42:03.086546 16146 ts_manager.cc:194] Registered new tserver with Master: ccc89d988dd64eebb9ecce0f7ed6302f (127.28.243.67:38383)
I20260809 22:42:03.087381 16146 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.67:59461
I20260809 22:42:03.095722 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:42:03.097486 16346 heartbeater.cc:503] Master 127.28.243.126:38511 was elected leader, sending a full tablet report...
I20260809 22:42:03.106755 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.100390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.28.243.126:38511)
0809 22:42:03.100651 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.100655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.101111 (+ 456us) server_negotiation.cc:407] Connection header received
0809 22:42:03.101226 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.101229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.101288 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.101368 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:03.101866 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.102596 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.104303 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.104821 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.104922 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.106102 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:03.106106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:03.106110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:03.106128 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:03.106159 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:03.106171 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:03.106245 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.106427 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.106510 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:42:03.111189 16146 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:42:03.118789 16146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:42:03.147550 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.132361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58524 (local address 127.28.243.66:34585)
0809 22:42:03.133077 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.133087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.133794 (+ 707us) server_negotiation.cc:407] Connection header received
0809 22:42:03.134249 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.134256 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.134524 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.134735 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:03.135899 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.138058 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.142214 (+ 4156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.145365 (+ 3151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.145506 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.145815 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.146671 (+ 856us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.146969 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":729,"threads_started":1}
I20260809 22:42:03.147547 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.133077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.28.243.65:45257)
0809 22:42:03.133862 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.133875 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.134509 (+ 634us) server_negotiation.cc:407] Connection header received
0809 22:42:03.134869 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.134880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.135163 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.135353 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:03.136541 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.138675 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.142506 (+ 3831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.144756 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.144931 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.145357 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.146628 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.146927 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":757,"threads_started":1}
I20260809 22:42:03.149003 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.132116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:34585 (local address 127.0.0.1:58524)
0809 22:42:03.133637 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.133685 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:03.133974 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:03.135086 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:03.135096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:03.135135 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:03.135644 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.135661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.138319 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.138328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.141831 (+ 3503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.141866 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.146126 (+ 4260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.146137 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.146351 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.146405 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 22:42:03.147924 (+ 1519us) client_negotiation.cc:240] Negotiation successful
0809 22:42:03.148395 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"mutex_wait_us":4,"thread_start_us":1934,"threads_started":1}
I20260809 22:42:03.149832 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.132455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37332)
0809 22:42:03.134400 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.134419 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:03.134639 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:03.135714 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:03.135722 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:03.135758 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:03.136288 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.136304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.138923 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.138931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.142176 (+ 3245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.142199 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.145615 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.145625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.146147 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.146254 (+ 107us) client_negotiation.cc:769] Sending connection context
0809 22:42:03.148074 (+ 1820us) client_negotiation.cc:240] Negotiation successful
0809 22:42:03.149283 (+ 1209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1774,"mutex_wait_us":904,"thread_start_us":781,"threads_started":1}
I20260809 22:42:03.155458 16414 tablet_service.cc:1467] Processing CreateTablet for tablet 861c4ef243984615bed9759ff9d6ca6d (DEFAULT_TABLE table=test-table [id=ab13b972f4fd4cb08491b783bbd510ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:03.155385 16275 tablet_service.cc:1467] Processing CreateTablet for tablet 861c4ef243984615bed9759ff9d6ca6d (DEFAULT_TABLE table=test-table [id=ab13b972f4fd4cb08491b783bbd510ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:03.157174 16275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861c4ef243984615bed9759ff9d6ca6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:03.157203 16414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861c4ef243984615bed9759ff9d6ca6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:03.157801 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.134446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:38383 (local address 127.0.0.1:48216)
0809 22:42:03.148740 (+ 14294us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.148761 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:03.148914 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:03.150039 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:03.150045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:03.150076 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:03.150820 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.150841 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.153642 (+ 2801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.153650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.155144 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.155160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.156817 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.156825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.157036 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.157077 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 22:42:03.157200 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 22:42:03.157471 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14140,"thread_start_us":13926,"threads_started":1}
I20260809 22:42:03.157819 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.134868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.28.243.67:38383)
0809 22:42:03.148916 (+ 14048us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.148929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.149001 (+ 72us) server_negotiation.cc:407] Connection header received
0809 22:42:03.149274 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.149284 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.149740 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.150198 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:03.151441 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.153431 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.155395 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.156506 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.156722 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.157030 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.157247 (+ 217us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.157421 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13513,"thread_start_us":13728,"threads_started":1}
I20260809 22:42:03.161795 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 861c4ef243984615bed9759ff9d6ca6d (DEFAULT_TABLE table=test-table [id=ab13b972f4fd4cb08491b783bbd510ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:03.163232 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861c4ef243984615bed9759ff9d6ca6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:03.174896 16638 tablet_bootstrap.cc:492] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: Bootstrap starting.
I20260809 22:42:03.175884 16637 tablet_bootstrap.cc:492] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Bootstrap starting.
I20260809 22:42:03.177884 16639 tablet_bootstrap.cc:492] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Bootstrap starting.
I20260809 22:42:03.180982 16637 tablet_bootstrap.cc:654] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:03.181097 16638 tablet_bootstrap.cc:654] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:03.181720 16639 tablet_bootstrap.cc:654] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:03.182209 16637 log.cc:824] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:03.182518 16638 log.cc:824] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:03.182998 16639 log.cc:824] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:03.186329 16638 tablet_bootstrap.cc:492] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: No bootstrap required, opened a new log
I20260809 22:42:03.186509 16637 tablet_bootstrap.cc:492] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: No bootstrap required, opened a new log
I20260809 22:42:03.186677 16638 ts_tablet_manager.cc:1397] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260809 22:42:03.186784 16637 ts_tablet_manager.cc:1397] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260809 22:42:03.191391 16639 tablet_bootstrap.cc:492] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: No bootstrap required, opened a new log
I20260809 22:42:03.191722 16639 ts_tablet_manager.cc:1397] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260809 22:42:03.195598 16638 raft_consensus.cc:348] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.196115 16638 raft_consensus.cc:374] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:03.196213 16638 raft_consensus.cc:729] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5ccaca61624917a5ca1ac7f495da4e, State: Initialized, Role: FOLLOWER
I20260809 22:42:03.197208 16637 raft_consensus.cc:348] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.197727 16637 raft_consensus.cc:374] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:03.197837 16637 raft_consensus.cc:729] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d32d14c8670430a9c959df5aa4baeee, State: Initialized, Role: FOLLOWER
I20260809 22:42:03.198504 16637 consensus_queue.cc:260] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.198649 16638 consensus_queue.cc:260] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.199935 16482 heartbeater.cc:503] Master 127.28.243.126:38511 was elected leader, sending a full tablet report...
I20260809 22:42:03.202245 16637 ts_tablet_manager.cc:1428] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260809 22:42:03.203262 16639 raft_consensus.cc:348] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.203800 16639 raft_consensus.cc:374] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:03.203912 16639 raft_consensus.cc:729] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc89d988dd64eebb9ecce0f7ed6302f, State: Initialized, Role: FOLLOWER
I20260809 22:42:03.204123 16638 ts_tablet_manager.cc:1428] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260809 22:42:03.204743 16639 consensus_queue.cc:260] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.206313 16618 heartbeater.cc:503] Master 127.28.243.126:38511 was elected leader, sending a full tablet report...
I20260809 22:42:03.206923 16639 ts_tablet_manager.cc:1428] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Time spent starting tablet: real 0.015s user 0.017s sys 0.000s
W20260809 22:42:03.320372 16347 tablet.cc:2369] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:42:03.322599 16619 tablet.cc:2369] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:42:03.325461 16483 tablet.cc:2369] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:03.375312 16645 raft_consensus.cc:482] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:42:03.375891 16645 raft_consensus.cc:504] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.377838 16645 leader_election.cc:288] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d32d14c8670430a9c959df5aa4baeee (127.28.243.66:34585), ccc89d988dd64eebb9ecce0f7ed6302f (127.28.243.67:38383)
I20260809 22:42:03.385699 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.378434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:54425 (local address 127.28.243.66:34585)
0809 22:42:03.378681 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.378686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.379338 (+ 652us) server_negotiation.cc:407] Connection header received
0809 22:42:03.379609 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.379617 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.379738 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.379847 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:03.380674 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.381521 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.383710 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.384693 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.384892 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.384984 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.385270 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.385405 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 22:42:03.385752 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.378469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:34585 (local address 127.28.243.65:54425)
0809 22:42:03.379212 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.379231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:03.379433 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:03.380000 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:03.380004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:03.380030 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:03.380463 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.380473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.381662 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.381672 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.383473 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.383487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.384889 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.384896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.385082 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.385108 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 22:42:03.385223 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 22:42:03.385365 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":337,"threads_started":1}
I20260809 22:42:03.386238 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.379685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:45487 (local address 127.28.243.67:38383)
0809 22:42:03.379889 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.379898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.380201 (+ 303us) server_negotiation.cc:407] Connection header received
0809 22:42:03.380439 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.380445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.380520 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.380676 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:03.381287 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.382072 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.384144 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.385264 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.385441 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.385520 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.385843 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.385989 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 22:42:03.386298 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.379347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:38383 (local address 127.28.243.65:45487)
0809 22:42:03.380085 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.380113 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:03.380269 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:03.380692 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:03.380698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:03.380725 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:03.381104 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.381112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.382235 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.382240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.383948 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:03.383961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.385495 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.385502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:03.385670 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.385694 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 22:42:03.385815 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 22:42:03.385956 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":365,"threads_started":1}
I20260809 22:42:03.387319 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861c4ef243984615bed9759ff9d6ca6d" candidate_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" is_pre_election: true
I20260809 22:42:03.387318 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861c4ef243984615bed9759ff9d6ca6d" candidate_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d32d14c8670430a9c959df5aa4baeee" is_pre_election: true
I20260809 22:42:03.387879 16571 raft_consensus.cc:2454] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5ccaca61624917a5ca1ac7f495da4e in term 0.
I20260809 22:42:03.387880 16434 raft_consensus.cc:2454] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a5ccaca61624917a5ca1ac7f495da4e in term 0.
I20260809 22:42:03.388737 16228 leader_election.cc:302] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a5ccaca61624917a5ca1ac7f495da4e, ccc89d988dd64eebb9ecce0f7ed6302f; no voters:
I20260809 22:42:03.389302 16645 raft_consensus.cc:2781] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:42:03.389497 16645 raft_consensus.cc:482] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:42:03.389709 16645 raft_consensus.cc:3037] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:03.392305 16645 raft_consensus.cc:504] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.393258 16645 leader_election.cc:288] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [CANDIDATE]: Term 1 election: Requested vote from peers 4d32d14c8670430a9c959df5aa4baeee (127.28.243.66:34585), ccc89d988dd64eebb9ecce0f7ed6302f (127.28.243.67:38383)
I20260809 22:42:03.393959 16434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861c4ef243984615bed9759ff9d6ca6d" candidate_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d32d14c8670430a9c959df5aa4baeee"
I20260809 22:42:03.393963 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861c4ef243984615bed9759ff9d6ca6d" candidate_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f"
I20260809 22:42:03.394188 16434 raft_consensus.cc:3037] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:03.394224 16571 raft_consensus.cc:3037] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:03.397778 16571 raft_consensus.cc:2454] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5ccaca61624917a5ca1ac7f495da4e in term 1.
I20260809 22:42:03.398101 16434 raft_consensus.cc:2454] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a5ccaca61624917a5ca1ac7f495da4e in term 1.
I20260809 22:42:03.398634 16228 leader_election.cc:302] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d32d14c8670430a9c959df5aa4baeee, 8a5ccaca61624917a5ca1ac7f495da4e; no voters:
I20260809 22:42:03.399191 16645 raft_consensus.cc:2781] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:03.399866 16645 raft_consensus.cc:686] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [term 1 LEADER]: Becoming Leader. State: Replica: 8a5ccaca61624917a5ca1ac7f495da4e, State: Running, Role: LEADER
I20260809 22:42:03.400470 16645 consensus_queue.cc:237] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } }
I20260809 22:42:03.407186 16144 catalog_manager.cc:5495] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a5ccaca61624917a5ca1ac7f495da4e (127.28.243.65). New cstate: current_term: 1 leader_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a5ccaca61624917a5ca1ac7f495da4e" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 45257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 } health_report { overall_health: UNKNOWN } } }
I20260809 22:42:03.468154 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.461821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.28.243.65:45257)
0809 22:42:03.462120 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.462125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.462146 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:42:03.462247 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.462251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.462365 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.462514 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:42:03.462984 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.463767 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.465389 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.466032 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.466151 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.466689 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:42:03.467355 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:42:03.467491 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.467701 (+ 210us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.467781 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260809 22:42:03.477233 16571 raft_consensus.cc:1261] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 1 FOLLOWER]: Refusing update from remote peer 8a5ccaca61624917a5ca1ac7f495da4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:03.477358 16434 raft_consensus.cc:1261] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 1 FOLLOWER]: Refusing update from remote peer 8a5ccaca61624917a5ca1ac7f495da4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:03.478281 16649 consensus_queue.cc:1035] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc89d988dd64eebb9ecce0f7ed6302f" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:03.478807 16645 consensus_queue.cc:1035] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d32d14c8670430a9c959df5aa4baeee" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:03.489512 16652 mvcc.cc:205] Tried to move back new op lower bound from 7316747564952309760 to 7316747564652756992. Current Snapshot: MvccSnapshot[applied={T|T < 7316747564952309760}]
I20260809 22:42:03.490447 16654 mvcc.cc:205] Tried to move back new op lower bound from 7316747564952309760 to 7316747564652756992. Current Snapshot: MvccSnapshot[applied={T|T < 7316747564952309760}]
I20260809 22:42:03.500540 16653 mvcc.cc:205] Tried to move back new op lower bound from 7316747564952309760 to 7316747564652756992. Current Snapshot: MvccSnapshot[applied={T|T < 7316747564952309760}]
I20260809 22:42:03.514039 16144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:03.514391 16144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:03.518985 16144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 551e4255d3104ccdb22d19cd24ee0fe0: Sending DeleteTablet for 3 replicas of tablet 861c4ef243984615bed9759ff9d6ca6d
I20260809 22:42:03.520819 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet 861c4ef243984615bed9759ff9d6ca6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:03 UTC) from {username='slave'} at 127.0.0.1:48216
I20260809 22:42:03.520941 16275 tablet_service.cc:1514] Processing DeleteTablet for tablet 861c4ef243984615bed9759ff9d6ca6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:03 UTC) from {username='slave'} at 127.0.0.1:37332
I20260809 22:42:03.521066 16414 tablet_service.cc:1514] Processing DeleteTablet for tablet 861c4ef243984615bed9759ff9d6ca6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:03 UTC) from {username='slave'} at 127.0.0.1:58524
I20260809 22:42:03.521981 16662 tablet_replica.cc:330] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: stopping tablet replica
I20260809 22:42:03.522372 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 16209
I20260809 22:42:03.522538 16662 raft_consensus.cc:2229] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:03.523182 16662 raft_consensus.cc:2258] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:03.525795 16662 ts_tablet_manager.cc:1905] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:03.534232 16662 ts_tablet_manager.cc:1918] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:03.534454 16662 log.cc:1196] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/861c4ef243984615bed9759ff9d6ca6d
I20260809 22:42:03.535121 16662 ts_tablet_manager.cc:1939] T 861c4ef243984615bed9759ff9d6ca6d P ccc89d988dd64eebb9ecce0f7ed6302f: Deleting consensus metadata
I20260809 22:42:03.537158 16663 tablet_replica.cc:330] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: stopping tablet replica
I20260809 22:42:03.537721 16663 raft_consensus.cc:2229] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:03.537717 16132 catalog_manager.cc:4841] TS ccc89d988dd64eebb9ecce0f7ed6302f (127.28.243.67:38383): tablet 861c4ef243984615bed9759ff9d6ca6d (table test-table [id=ab13b972f4fd4cb08491b783bbd510ca]) successfully deleted
I20260809 22:42:03.538334 16663 raft_consensus.cc:2258] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:03.540807 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.524130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.28.243.65:45257)
0809 22:42:03.524320 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.524329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.529963 (+ 5634us) server_negotiation.cc:407] Connection header received
0809 22:42:03.530188 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.530197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.530375 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.530508 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:03.531443 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.533441 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.536390 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.537218 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.537336 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.539166 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:03.539175 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:03.539189 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:03.539826 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:03.539876 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:03.539897 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:03.540047 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.540192 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.540318 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 22:42:03.540920 16663 ts_tablet_manager.cc:1905] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:03.541574 16341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:03.549029 16663 ts_tablet_manager.cc:1918] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:03.549235 16663 log.cc:1196] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/861c4ef243984615bed9759ff9d6ca6d
I20260809 22:42:03.549814 16663 ts_tablet_manager.cc:1939] T 861c4ef243984615bed9759ff9d6ca6d P 4d32d14c8670430a9c959df5aa4baeee: Deleting consensus metadata
I20260809 22:42:03.551637 16132 catalog_manager.cc:4841] TS 4d32d14c8670430a9c959df5aa4baeee (127.28.243.66:34585): tablet 861c4ef243984615bed9759ff9d6ca6d (table test-table [id=ab13b972f4fd4cb08491b783bbd510ca]) successfully deleted
I20260809 22:42:03.646016 16664 tablet_replica.cc:330] T 861c4ef243984615bed9759ff9d6ca6d P 8a5ccaca61624917a5ca1ac7f495da4e: stopping tablet replica
I20260809 22:42:03.646135 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 16209
W20260809 22:42:03.667205 16131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.65:45257 (error 108)
I20260809 22:42:03.667420 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 16350
I20260809 22:42:03.669222 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.668341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37362)
0809 22:42:03.668843 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.669007 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":274,"threads_started":1}
W20260809 22:42:03.669581 16131 catalog_manager.cc:4584] TS 8a5ccaca61624917a5ca1ac7f495da4e (127.28.243.65:45257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 861c4ef243984615bed9759ff9d6ca6d: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
I20260809 22:42:03.674775 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.668270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.28.243.66:34585)
0809 22:42:03.668508 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.668515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.668542 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:42:03.668784 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.668790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.668917 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.669037 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:03.669449 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.670250 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.671572 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.672009 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.672131 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.673268 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:03.673276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:03.673291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:03.673877 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:03.673933 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:03.673953 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:03.674060 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.674280 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.674374 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 22:42:03.675562 16476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:03.723490 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.722921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37376)
0809 22:42:03.723182 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.723300 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 22:42:03.724753 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.724288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37380)
0809 22:42:03.724474 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.724591 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 22:42:03.768230 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 16350
I20260809 22:42:03.785884 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.785252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37390)
0809 22:42:03.785527 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.785667 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 22:42:03.787281 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.786732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37404)
0809 22:42:03.786982 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.787081 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 22:42:03.789214 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 16486
I20260809 22:42:03.796245 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.790046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.28.243.67:38383)
0809 22:42:03.790250 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:03.790257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:03.790278 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:42:03.790478 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:03.790484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:03.790596 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:03.790710 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:03.791131 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.791848 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.793172 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:03.793570 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:03.793713 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:03.794838 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:03.794848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:03.794864 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:03.795459 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:03.795507 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:03.795529 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:03.795660 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:03.795796 (+ 136us) server_negotiation.cc:299] Negotiation successful
0809 22:42:03.795909 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:42:03.797075 16613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:03.868039 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.867408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37412)
0809 22:42:03.867727 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.867842 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260809 22:42:03.869361 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:03.868852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:45257 (local address 127.0.0.1:37424)
0809 22:42:03.869100 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:03.869195 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:45257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260809 22:42:03.889515 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 16486
I20260809 22:42:03.908600 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 16113
I20260809 22:42:03.909780 16178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:04.023263 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 16113
2026-08-09T22:42:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3186 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:04 dist-test-slave-rgb1 krb5kdc[16674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:04 dist-test-slave-rgb1 krb5kdc[16674](info): set up 2 sockets
Aug 09 22:42:04 dist-test-slave-rgb1 krb5kdc[16674](info): commencing operation
krb5kdc: starting...
W20260809 22:42:05.222901 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:05 dist-test-slave-rgb1 krb5kdc[16674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:05.569046 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39839
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:39839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:05.935410 16687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:05.935738 16687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:05.935832 16687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:05.943387 16687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:05.943511 16687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:05.943570 16687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:05.943631 16687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:05.943678 16687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:05.954013 16687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:39839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:05.955340 16687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:05.956694 16687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:05.969705 16692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:05.969758 16693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:05.969754 16695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:05.970911 16687 server_base.cc:1034] running on GCE node
I20260809 22:42:05.971902 16687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:05.973592 16687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:05.974843 16687 hybrid_clock.cc:648] HybridClock initialized: now 1786315325974779 us; error 55 us; skew 500 ppm
I20260809 22:42:05.975225 16687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:05 dist-test-slave-rgb1 krb5kdc[16674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315325, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:05.982798 16687 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:42:05.984647 16687 webserver.cc:460] Webserver started at http://127.28.243.126:36275/ using document root <none> and password file <none>
I20260809 22:42:05.985375 16687 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:05.985538 16687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:05.985877 16687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:05.989055 16687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "57c196ce00a84b2db8849ddc08874c9f"
format_stamp: "Formatted at 2026-08-09 22:42:05 on dist-test-slave-rgb1"
server_key: "61f3764df52a876562564411a87c7f4d"
server_key_iv: "e89e334d31d9ef272e38c220e78e4c91"
server_key_version: "encryptionkey@0"
I20260809 22:42:05.989846 16687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "57c196ce00a84b2db8849ddc08874c9f"
format_stamp: "Formatted at 2026-08-09 22:42:05 on dist-test-slave-rgb1"
server_key: "61f3764df52a876562564411a87c7f4d"
server_key_iv: "e89e334d31d9ef272e38c220e78e4c91"
server_key_version: "encryptionkey@0"
I20260809 22:42:05.995508 16687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 22:42:05.999202 16703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:06.000995 16687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:06.001205 16687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "57c196ce00a84b2db8849ddc08874c9f"
format_stamp: "Formatted at 2026-08-09 22:42:05 on dist-test-slave-rgb1"
server_key: "61f3764df52a876562564411a87c7f4d"
server_key_iv: "e89e334d31d9ef272e38c220e78e4c91"
server_key_version: "encryptionkey@0"
I20260809 22:42:06.001433 16687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:06.029706 16687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:06.034996 16687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:06.035359 16687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:06.057421 16687 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:39839
I20260809 22:42:06.057461 16756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:39839 every 8 connection(s)
I20260809 22:42:06.058981 16687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:06.062935 16757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:06.064173 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 16687
I20260809 22:42:06.064388 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:06.064934 29645 external_mini_cluster.cc:1428] Setting key 4bd95c67df00ad4f487c6e3b82565567
I20260809 22:42:06.078130 16757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f: Bootstrap starting.
Aug 09 22:42:06 dist-test-slave-rgb1 krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:06.082427 16757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:06.083850 16757 log.cc:824] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:06.087098 16757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f: No bootstrap required, opened a new log
I20260809 22:42:06.091401 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:06.067239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.28.243.126:39839)
0809 22:42:06.068148 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:06.068165 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:06.068217 (+ 52us) server_negotiation.cc:407] Connection header received
0809 22:42:06.070013 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:06.070044 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:06.070580 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:06.071083 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:06.072672 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:06.074849 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:06.076375 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:06.077463 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:06.077654 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:06.082964 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:06.082980 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:06.083001 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:06.083057 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:06.086670 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:06.087560 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:06.087567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:06.087573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:06.087758 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:06.088183 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:06.088193 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:06.088201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:06.088872 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:06.089268 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:06.089797 (+ 529us) server_negotiation.cc:299] Negotiation successful
0809 22:42:06.090273 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":377,"threads_started":1}
I20260809 22:42:06.096323 16757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c196ce00a84b2db8849ddc08874c9f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39839 } }
I20260809 22:42:06.096840 16757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:06.096978 16757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c196ce00a84b2db8849ddc08874c9f, State: Initialized, Role: FOLLOWER
I20260809 22:42:06.097700 16757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c196ce00a84b2db8849ddc08874c9f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39839 } }
I20260809 22:42:06.097937 16757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:06.098109 16757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:06.098306 16757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:06.101428 16757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c196ce00a84b2db8849ddc08874c9f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39839 } }
I20260809 22:42:06.102020 16757 leader_election.cc:302] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c196ce00a84b2db8849ddc08874c9f; no voters:
I20260809 22:42:06.102773 16757 leader_election.cc:288] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:06.102995 16762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:06.103878 16762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [term 1 LEADER]: Becoming Leader. State: Replica: 57c196ce00a84b2db8849ddc08874c9f, State: Running, Role: LEADER
I20260809 22:42:06.104472 16762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c196ce00a84b2db8849ddc08874c9f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39839 } }
I20260809 22:42:06.105432 16757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:06.108049 16763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c196ce00a84b2db8849ddc08874c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c196ce00a84b2db8849ddc08874c9f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39839 } } }
I20260809 22:42:06.108611 16763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:06.109182 16765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c196ce00a84b2db8849ddc08874c9f. Latest consensus state: current_term: 1 leader_uuid: "57c196ce00a84b2db8849ddc08874c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c196ce00a84b2db8849ddc08874c9f" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 39839 } } }
I20260809 22:42:06.109655 16765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:06.111155 16768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:06.117017 16768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:06.130728 16768 catalog_manager.cc:1347] Generated new cluster ID: d9535392d8104ce595b262eda9d2b7cb
I20260809 22:42:06.130888 16768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:06.144243 16768 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:06.146005 16768 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:06.156867 16768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f: Generated new TSK 0
I20260809 22:42:06.157883 16768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:42:06.249228 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39839
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:06.659277 16786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:06.659588 16786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:06.659770 16786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:06.667366 16786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:06.667580 16786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:06.667780 16786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:42:06.678108 16786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:06.679507 16786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:06.680904 16786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:06.694840 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:06.694927 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:06.695048 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:06.696522 16786 server_base.cc:1034] running on GCE node
I20260809 22:42:06.697358 16786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:06.698468 16786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:06.699774 16786 hybrid_clock.cc:648] HybridClock initialized: now 1786315326699728 us; error 29 us; skew 500 ppm
I20260809 22:42:06.700177 16786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:06 dist-test-slave-rgb1 krb5kdc[16674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:06.704120 16786 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:42:06.706177 16786 webserver.cc:460] Webserver started at http://127.28.243.65:36317/ using document root <none> and password file <none>
I20260809 22:42:06.707046 16786 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:06.707203 16786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:06.707549 16786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:06.710776 16786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "162acfb15aeb4e989a9af70fe66c7234"
format_stamp: "Formatted at 2026-08-09 22:42:06 on dist-test-slave-rgb1"
server_key: "a3f7bc82b5cd4de3797e3233345b861d"
server_key_iv: "ff35e4db1241251830725fe6bdd3f184"
server_key_version: "encryptionkey@0"
I20260809 22:42:06.711697 16786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "162acfb15aeb4e989a9af70fe66c7234"
format_stamp: "Formatted at 2026-08-09 22:42:06 on dist-test-slave-rgb1"
server_key: "a3f7bc82b5cd4de3797e3233345b861d"
server_key_iv: "ff35e4db1241251830725fe6bdd3f184"
server_key_version: "encryptionkey@0"
I20260809 22:42:06.717356 16786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 22:42:06.721339 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:06.722985 16786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 22:42:06.723219 16786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "162acfb15aeb4e989a9af70fe66c7234"
format_stamp: "Formatted at 2026-08-09 22:42:06 on dist-test-slave-rgb1"
server_key: "a3f7bc82b5cd4de3797e3233345b861d"
server_key_iv: "ff35e4db1241251830725fe6bdd3f184"
server_key_version: "encryptionkey@0"
I20260809 22:42:06.723476 16786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:06.736212 16786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:06.741349 16786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:06.741827 16786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:06.743297 16786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:06.744962 16786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:06.745074 16786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:06.745239 16786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:06.745321 16786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:06.785650 16786 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:34975
I20260809 22:42:06.785681 16918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:34975 every 8 connection(s)
I20260809 22:42:06.787220 16786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:42:06.795233 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 16786
I20260809 22:42:06.795481 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:42:06.795954 29645 external_mini_cluster.cc:1428] Setting key 89dd96a89fe767c9535418191e71ac37
Aug 09 22:42:06 dist-test-slave-rgb1 krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315326, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:06.811980 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:06.790449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:37343 (local address 127.28.243.126:39839)
0809 22:42:06.790970 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:06.790976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:06.792079 (+ 1103us) server_negotiation.cc:407] Connection header received
0809 22:42:06.794110 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:06.794116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:06.794223 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:06.794353 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:06.796134 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:06.797134 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:06.799606 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:06.800073 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:06.800189 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:06.805386 (+ 5197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:06.805392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:06.805395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:06.805431 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:06.808461 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:06.809432 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:06.809438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:06.809442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:06.809593 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:06.810082 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:06.810087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:06.810090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:06.810433 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:06.810607 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:06.811343 (+ 736us) server_negotiation.cc:299] Negotiation successful
0809 22:42:06.811601 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":270,"threads_started":1}
I20260809 22:42:06.813230 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:06.791046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39839 (local address 127.28.243.65:37343)
0809 22:42:06.791823 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:06.791878 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:06.793811 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:06.794623 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:06.794632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:06.795241 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:06.795938 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:06.795956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:06.797311 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:06.797315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:06.799397 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:06.799424 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:06.800339 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:06.800349 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:06.800734 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:06.801711 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:42:06.801749 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:06.805019 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:06.808710 (+ 3691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:06.808720 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:06.808740 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:06.809286 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:06.809719 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:06.809724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:06.809727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:06.809946 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:06.810584 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:06.810592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:06.810966 (+ 374us) client_negotiation.cc:769] Sending connection context
0809 22:42:06.811260 (+ 294us) client_negotiation.cc:240] Negotiation successful
0809 22:42:06.811854 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":313,"threads_started":1}
I20260809 22:42:06.814951 16920 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39839
I20260809 22:42:06.815321 16920 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:06.815995 16920 heartbeater.cc:511] Master 127.28.243.126:39839 requested a full tablet report, sending...
I20260809 22:42:06.818408 16720 ts_manager.cc:194] Registered new tserver with Master: 162acfb15aeb4e989a9af70fe66c7234 (127.28.243.65:34975)
I20260809 22:42:06.820793 16720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:37343
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:42:06.873582 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39839
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:07.245265 16928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:07.245589 16928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:07.245688 16928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:07.253196 16928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:07.253332 16928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:07.253453 16928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:42:07.263492 16928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:07.264844 16928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:07.266108 16928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:07.280143 16937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:07.280102 16935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:07.280194 16934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:07.286830 16928 server_base.cc:1034] running on GCE node
I20260809 22:42:07.287595 16928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:07.288770 16928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:07.290040 16928 hybrid_clock.cc:648] HybridClock initialized: now 1786315327289976 us; error 79 us; skew 500 ppm
I20260809 22:42:07.290391 16928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:07 dist-test-slave-rgb1 krb5kdc[16674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:07.293855 16928 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:42:07.295547 16928 webserver.cc:460] Webserver started at http://127.28.243.66:42903/ using document root <none> and password file <none>
I20260809 22:42:07.296322 16928 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:07.296487 16928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:07.296849 16928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:07.300284 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "49fce04acb3e4beca3d70565479edfc4"
format_stamp: "Formatted at 2026-08-09 22:42:07 on dist-test-slave-rgb1"
server_key: "fd5fbadd128a8ce334fb356a6b1e19f7"
server_key_iv: "f6cbfbca98dbcdd6aba218cae571a794"
server_key_version: "encryptionkey@0"
I20260809 22:42:07.301332 16928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "49fce04acb3e4beca3d70565479edfc4"
format_stamp: "Formatted at 2026-08-09 22:42:07 on dist-test-slave-rgb1"
server_key: "fd5fbadd128a8ce334fb356a6b1e19f7"
server_key_iv: "f6cbfbca98dbcdd6aba218cae571a794"
server_key_version: "encryptionkey@0"
I20260809 22:42:07.308079 16928 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 22:42:07.311959 16944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:07.313676 16928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:07.313879 16928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "49fce04acb3e4beca3d70565479edfc4"
format_stamp: "Formatted at 2026-08-09 22:42:07 on dist-test-slave-rgb1"
server_key: "fd5fbadd128a8ce334fb356a6b1e19f7"
server_key_iv: "f6cbfbca98dbcdd6aba218cae571a794"
server_key_version: "encryptionkey@0"
I20260809 22:42:07.314117 16928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:07.336923 16928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:07.341666 16928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:07.342020 16928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:07.343134 16928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:07.344606 16928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:07.344718 16928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:07.344882 16928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:07.344969 16928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:07.383744 16928 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:38645
I20260809 22:42:07.383859 17058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:38645 every 8 connection(s)
I20260809 22:42:07.385298 16928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:42:07.388736 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 16928
I20260809 22:42:07.388971 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:42:07.389617 29645 external_mini_cluster.cc:1428] Setting key d77590f738a0a6c91ed11f40413433dd
Aug 09 22:42:07 dist-test-slave-rgb1 krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:07.414669 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:07.388268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:42687 (local address 127.28.243.126:39839)
0809 22:42:07.388954 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:07.388973 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:07.390006 (+ 1033us) server_negotiation.cc:407] Connection header received
0809 22:42:07.392342 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:07.392347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:07.392425 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:07.392527 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:07.394901 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.396361 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.399703 (+ 3342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.400532 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.400732 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:07.406719 (+ 5987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:07.406725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:07.406728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:07.406766 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:07.410376 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:07.411616 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:07.411621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:07.411624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:07.411820 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:07.412474 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:07.412478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:07.412480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:07.412876 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:07.413056 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:07.414163 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0809 22:42:07.414365 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":384,"threads_started":1}
I20260809 22:42:07.416457 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:07.388850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39839 (local address 127.28.243.66:42687)
0809 22:42:07.389786 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:07.389837 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:07.392004 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:07.392867 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:07.392878 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:07.393743 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:07.394668 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:07.394701 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.396643 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.396654 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:07.399393 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:07.399438 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.401153 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.401164 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:07.401519 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:07.403139 (+ 1620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:42:07.403209 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:07.406370 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:07.410584 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:07.410599 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:07.410624 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:07.411419 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:07.411989 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:07.411997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:07.412004 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:07.412311 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:07.413024 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:07.413037 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:07.413604 (+ 567us) client_negotiation.cc:769] Sending connection context
0809 22:42:07.414025 (+ 421us) client_negotiation.cc:240] Negotiation successful
0809 22:42:07.414716 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":348,"threads_started":1}
I20260809 22:42:07.418351 17059 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39839
I20260809 22:42:07.418720 17059 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:07.419571 17059 heartbeater.cc:511] Master 127.28.243.126:39839 requested a full tablet report, sending...
I20260809 22:42:07.421403 16720 ts_manager.cc:194] Registered new tserver with Master: 49fce04acb3e4beca3d70565479edfc4 (127.28.243.66:38645)
I20260809 22:42:07.422376 16720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:42687
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:42:07.477643 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39839
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:42:07.824477 16920 heartbeater.cc:503] Master 127.28.243.126:39839 was elected leader, sending a full tablet report...
W20260809 22:42:07.842324 17068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:07.842689 17068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:07.842873 17068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:07.850602 17068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:07.850786 17068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:07.850966 17068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:42:07.861442 17068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:39839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:07.862855 17068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:07.864265 17068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:07.878836 17073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:07.878796 17074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:07.878929 17076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:07.879700 17068 server_base.cc:1034] running on GCE node
I20260809 22:42:07.880344 17068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:07.881469 17068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:07.882722 17068 hybrid_clock.cc:648] HybridClock initialized: now 1786315327882628 us; error 89 us; skew 500 ppm
I20260809 22:42:07.883070 17068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:07 dist-test-slave-rgb1 krb5kdc[16674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:07.886706 17068 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:42:07.888422 17068 webserver.cc:460] Webserver started at http://127.28.243.67:42431/ using document root <none> and password file <none>
I20260809 22:42:07.889219 17068 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:07.889387 17068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:07.889740 17068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:07.892953 17068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "fddee107c2e74445a10ba947449c04de"
format_stamp: "Formatted at 2026-08-09 22:42:07 on dist-test-slave-rgb1"
server_key: "31f15fca47c53a1daa38651d9602d5e3"
server_key_iv: "6de3ae663663cf6c717b36e475a7da32"
server_key_version: "encryptionkey@0"
I20260809 22:42:07.893776 17068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "fddee107c2e74445a10ba947449c04de"
format_stamp: "Formatted at 2026-08-09 22:42:07 on dist-test-slave-rgb1"
server_key: "31f15fca47c53a1daa38651d9602d5e3"
server_key_iv: "6de3ae663663cf6c717b36e475a7da32"
server_key_version: "encryptionkey@0"
I20260809 22:42:07.899256 17068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260809 22:42:07.903280 17083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:07.904950 17068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:07.905270 17068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "fddee107c2e74445a10ba947449c04de"
format_stamp: "Formatted at 2026-08-09 22:42:07 on dist-test-slave-rgb1"
server_key: "31f15fca47c53a1daa38651d9602d5e3"
server_key_iv: "6de3ae663663cf6c717b36e475a7da32"
server_key_version: "encryptionkey@0"
I20260809 22:42:07.905563 17068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:07.927459 17068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:07.932312 17068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:07.932731 17068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:07.933938 17068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:07.935431 17068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:07.935544 17068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:07.935709 17068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:07.935789 17068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:07.975069 17068 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:37711
I20260809 22:42:07.975171 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:37711 every 8 connection(s)
I20260809 22:42:07.976490 17068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:42:07.984262 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 17068
I20260809 22:42:07.984534 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:42:07.985164 29645 external_mini_cluster.cc:1428] Setting key 1bdb75e06def103780124f37bc28ffc9
Aug 09 22:42:07 dist-test-slave-rgb1 krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315327, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:07.999410 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:07.979517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:44587 (local address 127.28.243.126:39839)
0809 22:42:07.980052 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:07.980060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:07.981137 (+ 1077us) server_negotiation.cc:407] Connection header received
0809 22:42:07.982987 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:07.982992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:07.983109 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:07.983214 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:07.985172 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.986182 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.988092 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.988740 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.988889 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:07.992807 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:07.992813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:07.992816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:07.992915 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:07.995925 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:07.996926 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:07.996930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:07.996933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:07.997084 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:07.997562 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:07.997566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:07.997568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:07.997880 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:07.998018 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:07.998736 (+ 718us) server_negotiation.cc:299] Negotiation successful
0809 22:42:07.998976 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":232,"threads_started":1}
I20260809 22:42:08.000717 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:07.980012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:39839 (local address 127.28.243.67:44587)
0809 22:42:07.980909 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:07.980954 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:07.982711 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:07.983476 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:07.983487 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:07.984157 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:07.984942 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:07.984963 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.986384 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.986387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:07.987926 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:07.987939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:07.989040 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:07.989047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:07.989260 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:07.990050 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:42:07.990080 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:07.992509 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:07.996113 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:07.996121 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:07.996135 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:07.996717 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:07.997205 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:07.997210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:07.997214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:07.997414 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:07.998013 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:07.998021 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:07.998371 (+ 350us) client_negotiation.cc:769] Sending connection context
0809 22:42:07.998636 (+ 265us) client_negotiation.cc:240] Negotiation successful
0809 22:42:07.999034 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":337,"threads_started":1}
I20260809 22:42:08.002490 17198 heartbeater.cc:347] Connected to a master server at 127.28.243.126:39839
I20260809 22:42:08.002784 17198 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:08.003526 17198 heartbeater.cc:511] Master 127.28.243.126:39839 requested a full tablet report, sending...
I20260809 22:42:08.005364 16719 ts_manager.cc:194] Registered new tserver with Master: fddee107c2e74445a10ba947449c04de (127.28.243.67:37711)
I20260809 22:42:08.006716 16719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:44587
I20260809 22:42:08.015304 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:42:08.036631 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.020330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.28.243.126:39839)
0809 22:42:08.020526 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.020531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.021123 (+ 592us) server_negotiation.cc:407] Connection header received
0809 22:42:08.021270 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.021273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.021353 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.021447 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:08.023223 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.024057 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.025714 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.026165 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.026288 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.030083 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:08.030088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:08.030092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:08.030129 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:08.033277 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.034237 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.034242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.034245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.034381 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.034998 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.035002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.035004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.035355 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:08.035613 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.036083 (+ 470us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.036248 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 22:42:08.041164 16719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:42:08.049808 16719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:42:08.076963 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.063498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:38645 (local address 127.0.0.1:40730)
0809 22:42:08.064766 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:08.064820 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:08.065023 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:08.066964 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:08.066973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:08.067007 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:08.067529 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.067542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.069991 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.069998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.073677 (+ 3679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.073693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.075870 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.075877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.076125 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.076185 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 22:42:08.076300 (+ 115us) client_negotiation.cc:240] Negotiation successful
0809 22:42:08.076510 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":1737,"threads_started":1}
I20260809 22:42:08.078578 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.063888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.28.243.66:38645)
0809 22:42:08.066274 (+ 2386us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.066283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.066311 (+ 28us) server_negotiation.cc:407] Connection header received
0809 22:42:08.066417 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.066425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.066679 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.067752 (+ 1073us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:08.068017 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.069801 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.073946 (+ 4145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.075621 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.077713 (+ 2092us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.077952 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.078023 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.078203 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2225,"thread_start_us":2115,"threads_started":1}
I20260809 22:42:08.083246 16993 tablet_service.cc:1467] Processing CreateTablet for tablet bb8f58f526d7444db1e3af69d89390ff (DEFAULT_TABLE table=test-table [id=54459106900f49e19638602bf1c67214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:08.085127 16993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8f58f526d7444db1e3af69d89390ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:08.091491 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.064492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.28.243.67:37711)
0809 22:42:08.079376 (+ 14884us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.079388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.079431 (+ 43us) server_negotiation.cc:407] Connection header received
0809 22:42:08.079532 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.079541 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.079794 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.080899 (+ 1105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:08.081258 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.083026 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.088454 (+ 5428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.090148 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.090291 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.090528 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.090930 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.091122 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14692,"thread_start_us":16846,"threads_started":1}
I20260809 22:42:08.091501 17214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.063904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:37711 (local address 127.0.0.1:60882)
0809 22:42:08.065663 (+ 1759us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:08.065681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:08.065796 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:08.080093 (+ 14297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:08.080099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:08.080133 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:08.080633 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.080651 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.083208 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.083217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.088185 (+ 4968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.088206 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.090427 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.090436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.090682 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.090726 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 22:42:08.090842 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 22:42:08.091015 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1619,"mutex_wait_us":1926,"thread_start_us":1717,"threads_started":1}
I20260809 22:42:08.093873 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.066660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.28.243.65:34975)
0809 22:42:08.085177 (+ 18517us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.085188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.085269 (+ 81us) server_negotiation.cc:407] Connection header received
0809 22:42:08.085428 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.085437 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.085739 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.086630 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:08.086921 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.088547 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.091192 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.092736 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.092888 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.093142 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.093368 (+ 226us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.093531 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18174,"thread_start_us":18000,"threads_started":1}
I20260809 22:42:08.093876 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.066047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:34975 (local address 127.0.0.1:44120)
0809 22:42:08.077390 (+ 11343us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:08.077412 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:08.077536 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:08.086026 (+ 8490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:08.086032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:08.086073 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:08.086400 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.086417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.089159 (+ 2742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.089165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.090998 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.091012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.092946 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.092955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.093152 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.093191 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 22:42:08.093323 (+ 132us) client_negotiation.cc:240] Negotiation successful
0809 22:42:08.093492 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11225,"mutex_wait_us":5,"thread_start_us":22838,"threads_started":1}
I20260809 22:42:08.096068 17131 tablet_service.cc:1467] Processing CreateTablet for tablet bb8f58f526d7444db1e3af69d89390ff (DEFAULT_TABLE table=test-table [id=54459106900f49e19638602bf1c67214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:08.097781 17131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8f58f526d7444db1e3af69d89390ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:08.097730 16852 tablet_service.cc:1467] Processing CreateTablet for tablet bb8f58f526d7444db1e3af69d89390ff (DEFAULT_TABLE table=test-table [id=54459106900f49e19638602bf1c67214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:08.099188 16852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb8f58f526d7444db1e3af69d89390ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:08.100692 17218 tablet_bootstrap.cc:492] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Bootstrap starting.
I20260809 22:42:08.104344 17218 tablet_bootstrap.cc:654] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:08.106042 17218 log.cc:824] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:08.113988 17220 tablet_bootstrap.cc:492] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Bootstrap starting.
I20260809 22:42:08.116065 17221 tablet_bootstrap.cc:492] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Bootstrap starting.
I20260809 22:42:08.118642 17218 tablet_bootstrap.cc:492] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: No bootstrap required, opened a new log
I20260809 22:42:08.118642 17220 tablet_bootstrap.cc:654] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:08.118978 17218 ts_tablet_manager.cc:1397] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20260809 22:42:08.119693 17221 tablet_bootstrap.cc:654] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:08.120043 17220 log.cc:824] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:08.120852 17221 log.cc:824] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:08.123694 17220 tablet_bootstrap.cc:492] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: No bootstrap required, opened a new log
I20260809 22:42:08.124138 17220 ts_tablet_manager.cc:1397] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260809 22:42:08.124297 17221 tablet_bootstrap.cc:492] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: No bootstrap required, opened a new log
I20260809 22:42:08.124615 17221 ts_tablet_manager.cc:1397] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260809 22:42:08.128064 17218 raft_consensus.cc:348] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.128497 17218 raft_consensus.cc:374] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:08.128703 17218 raft_consensus.cc:729] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49fce04acb3e4beca3d70565479edfc4, State: Initialized, Role: FOLLOWER
I20260809 22:42:08.129465 17218 consensus_queue.cc:260] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.132449 17059 heartbeater.cc:503] Master 127.28.243.126:39839 was elected leader, sending a full tablet report...
I20260809 22:42:08.134658 17218 ts_tablet_manager.cc:1428] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260809 22:42:08.135702 17221 raft_consensus.cc:348] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.135702 17220 raft_consensus.cc:348] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.136209 17221 raft_consensus.cc:374] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:08.136216 17220 raft_consensus.cc:374] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:08.136322 17221 raft_consensus.cc:729] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 162acfb15aeb4e989a9af70fe66c7234, State: Initialized, Role: FOLLOWER
I20260809 22:42:08.136328 17220 raft_consensus.cc:729] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddee107c2e74445a10ba947449c04de, State: Initialized, Role: FOLLOWER
I20260809 22:42:08.137066 17220 consensus_queue.cc:260] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.137066 17221 consensus_queue.cc:260] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
W20260809 22:42:08.137773 17061 tablet.cc:2369] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:08.141959 17198 heartbeater.cc:503] Master 127.28.243.126:39839 was elected leader, sending a full tablet report...
I20260809 22:42:08.142565 17220 ts_tablet_manager.cc:1428] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Time spent starting tablet: real 0.018s user 0.010s sys 0.009s
I20260809 22:42:08.143394 17221 ts_tablet_manager.cc:1428] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Time spent starting tablet: real 0.019s user 0.015s sys 0.002s
I20260809 22:42:08.150851 17224 raft_consensus.cc:482] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:42:08.151266 17224 raft_consensus.cc:504] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.153244 17224 leader_election.cc:288] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fddee107c2e74445a10ba947449c04de (127.28.243.67:37711), 162acfb15aeb4e989a9af70fe66c7234 (127.28.243.65:34975)
I20260809 22:42:08.163719 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.153754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:50605 (local address 127.28.243.67:37711)
0809 22:42:08.153980 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.153986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.154957 (+ 971us) server_negotiation.cc:407] Connection header received
0809 22:42:08.155434 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.155443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.155535 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.155648 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:08.156727 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.157910 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.161156 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.162428 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.162634 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.162699 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.163254 (+ 555us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.163439 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:42:08.163724 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.154958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:56871 (local address 127.28.243.65:34975)
0809 22:42:08.155198 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.155206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.155459 (+ 253us) server_negotiation.cc:407] Connection header received
0809 22:42:08.155545 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.155561 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.155673 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.155768 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:08.156749 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.157904 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.161156 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.162428 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.162619 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.162699 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.163254 (+ 555us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.163409 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:42:08.163915 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.154646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:34975 (local address 127.28.243.66:56871)
0809 22:42:08.155339 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:08.155353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:08.155445 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:08.155890 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:08.155896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:08.155926 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:08.156584 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.156598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.158083 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.158091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.160905 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.160915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.162737 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.162745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.163006 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.163042 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 22:42:08.163151 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 22:42:08.163358 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":233,"thread_start_us":361,"threads_started":1}
I20260809 22:42:08.164363 17227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.153753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:37711 (local address 127.28.243.66:50605)
0809 22:42:08.154802 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:08.154821 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:08.155110 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:08.155712 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:08.155715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:08.155861 (+ 146us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:08.156501 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.156515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.158105 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.158111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.160898 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:08.160915 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.162739 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.162744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:08.163066 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.163106 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 22:42:08.163184 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 22:42:08.163358 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":241,"thread_start_us":462,"threads_started":1}
I20260809 22:42:08.165141 16873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8f58f526d7444db1e3af69d89390ff" candidate_uuid: "49fce04acb3e4beca3d70565479edfc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162acfb15aeb4e989a9af70fe66c7234" is_pre_election: true
I20260809 22:42:08.165282 17152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8f58f526d7444db1e3af69d89390ff" candidate_uuid: "49fce04acb3e4beca3d70565479edfc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddee107c2e74445a10ba947449c04de" is_pre_election: true
I20260809 22:42:08.165820 17152 raft_consensus.cc:2454] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49fce04acb3e4beca3d70565479edfc4 in term 0.
I20260809 22:42:08.165793 16873 raft_consensus.cc:2454] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49fce04acb3e4beca3d70565479edfc4 in term 0.
I20260809 22:42:08.166546 16945 leader_election.cc:302] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fce04acb3e4beca3d70565479edfc4, fddee107c2e74445a10ba947449c04de; no voters:
I20260809 22:42:08.167037 17224 raft_consensus.cc:2781] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:42:08.167219 17224 raft_consensus.cc:482] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:42:08.167335 17224 raft_consensus.cc:3037] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:08.169826 17224 raft_consensus.cc:504] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.170377 17224 leader_election.cc:288] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [CANDIDATE]: Term 1 election: Requested vote from peers fddee107c2e74445a10ba947449c04de (127.28.243.67:37711), 162acfb15aeb4e989a9af70fe66c7234 (127.28.243.65:34975)
I20260809 22:42:08.171002 17152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8f58f526d7444db1e3af69d89390ff" candidate_uuid: "49fce04acb3e4beca3d70565479edfc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddee107c2e74445a10ba947449c04de"
I20260809 22:42:08.171226 17152 raft_consensus.cc:3037] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:08.171329 16873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb8f58f526d7444db1e3af69d89390ff" candidate_uuid: "49fce04acb3e4beca3d70565479edfc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "162acfb15aeb4e989a9af70fe66c7234"
I20260809 22:42:08.171589 16873 raft_consensus.cc:3037] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:08.173830 17152 raft_consensus.cc:2454] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49fce04acb3e4beca3d70565479edfc4 in term 1.
I20260809 22:42:08.174026 16873 raft_consensus.cc:2454] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49fce04acb3e4beca3d70565479edfc4 in term 1.
I20260809 22:42:08.174350 16945 leader_election.cc:302] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49fce04acb3e4beca3d70565479edfc4, fddee107c2e74445a10ba947449c04de; no voters:
I20260809 22:42:08.175012 17224 raft_consensus.cc:2781] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:08.175570 17224 raft_consensus.cc:686] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 1 LEADER]: Becoming Leader. State: Replica: 49fce04acb3e4beca3d70565479edfc4, State: Running, Role: LEADER
I20260809 22:42:08.176127 17224 consensus_queue.cc:237] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } }
I20260809 22:42:08.181613 16720 catalog_manager.cc:5495] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49fce04acb3e4beca3d70565479edfc4 (127.28.243.66). New cstate: current_term: 1 leader_uuid: "49fce04acb3e4beca3d70565479edfc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49fce04acb3e4beca3d70565479edfc4" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 38645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 } health_report { overall_health: UNKNOWN } } }
I20260809 22:42:08.206216 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.199812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.28.243.66:38645)
0809 22:42:08.200002 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.200006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.200026 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:42:08.200184 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.200188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.200276 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.200408 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:42:08.200888 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.201867 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.203516 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.204018 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.204132 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.204773 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:42:08.205382 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:42:08.205530 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.205809 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.205864 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 22:42:08.216149 17152 raft_consensus.cc:1261] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 1 FOLLOWER]: Refusing update from remote peer 49fce04acb3e4beca3d70565479edfc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:08.216250 16873 raft_consensus.cc:1261] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 1 FOLLOWER]: Refusing update from remote peer 49fce04acb3e4beca3d70565479edfc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:08.217522 17229 consensus_queue.cc:1035] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fddee107c2e74445a10ba947449c04de" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:08.218051 17224 consensus_queue.cc:1035] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "162acfb15aeb4e989a9af70fe66c7234" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 22:42:08.228339 17199 tablet.cc:2369] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:08.229154 17232 mvcc.cc:205] Tried to move back new op lower bound from 7316747584360525824 to 7316747584214933504. Current Snapshot: MvccSnapshot[applied={T|T < 7316747584360525824}]
I20260809 22:42:08.234189 17234 mvcc.cc:205] Tried to move back new op lower bound from 7316747584360525824 to 7316747584214933504. Current Snapshot: MvccSnapshot[applied={T|T < 7316747584360525824}]
I20260809 22:42:08.248510 16719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:08.248963 16719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:08.253583 16719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57c196ce00a84b2db8849ddc08874c9f: Sending DeleteTablet for 3 replicas of tablet bb8f58f526d7444db1e3af69d89390ff
I20260809 22:42:08.255038 16993 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8f58f526d7444db1e3af69d89390ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:08 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:40730
I20260809 22:42:08.255089 17131 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8f58f526d7444db1e3af69d89390ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:08 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:60882
I20260809 22:42:08.256814 16852 tablet_service.cc:1514] Processing DeleteTablet for tablet bb8f58f526d7444db1e3af69d89390ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:08 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:44120
I20260809 22:42:08.257548 17242 tablet_replica.cc:330] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: stopping tablet replica
I20260809 22:42:08.257900 17243 tablet_replica.cc:330] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: stopping tablet replica
I20260809 22:42:08.258209 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 16786
I20260809 22:42:08.258441 17242 raft_consensus.cc:2229] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:08.259016 17242 raft_consensus.cc:2258] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:08.259744 17243 raft_consensus.cc:2229] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:42:08.260517 17243 raft_consensus.cc:2258] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:08.262979 17242 ts_tablet_manager.cc:1905] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:08.267185 17243 ts_tablet_manager.cc:1905] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:08.270916 17242 ts_tablet_manager.cc:1918] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:08.271202 17242 log.cc:1196] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/bb8f58f526d7444db1e3af69d89390ff
I20260809 22:42:08.271839 17242 ts_tablet_manager.cc:1939] T bb8f58f526d7444db1e3af69d89390ff P fddee107c2e74445a10ba947449c04de: Deleting consensus metadata
I20260809 22:42:08.273085 17244 tablet_replica.cc:330] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: stopping tablet replica
I20260809 22:42:08.274181 16704 catalog_manager.cc:4841] TS fddee107c2e74445a10ba947449c04de (127.28.243.67:37711): tablet bb8f58f526d7444db1e3af69d89390ff (table test-table [id=54459106900f49e19638602bf1c67214]) successfully deleted
I20260809 22:42:08.274286 17244 raft_consensus.cc:2229] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:08.274873 17244 raft_consensus.cc:2258] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:08.276147 17243 ts_tablet_manager.cc:1918] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:08.276412 17243 log.cc:1196] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/bb8f58f526d7444db1e3af69d89390ff
I20260809 22:42:08.277097 17243 ts_tablet_manager.cc:1939] T bb8f58f526d7444db1e3af69d89390ff P 49fce04acb3e4beca3d70565479edfc4: Deleting consensus metadata
I20260809 22:42:08.277515 17244 ts_tablet_manager.cc:1905] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:08.279083 16706 catalog_manager.cc:4841] TS 49fce04acb3e4beca3d70565479edfc4 (127.28.243.66:38645): tablet bb8f58f526d7444db1e3af69d89390ff (table test-table [id=54459106900f49e19638602bf1c67214]) successfully deleted
Aug 09 22:42:08 dist-test-slave-rgb1 krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:42:08.284332 17244 ts_tablet_manager.cc:1918] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:08.284513 17244 log.cc:1196] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/bb8f58f526d7444db1e3af69d89390ff
I20260809 22:42:08.285058 17244 ts_tablet_manager.cc:1939] T bb8f58f526d7444db1e3af69d89390ff P 162acfb15aeb4e989a9af70fe66c7234: Deleting consensus metadata
I20260809 22:42:08.286937 16704 catalog_manager.cc:4841] TS 162acfb15aeb4e989a9af70fe66c7234 (127.28.243.65:34975): tablet bb8f58f526d7444db1e3af69d89390ff (table test-table [id=54459106900f49e19638602bf1c67214]) successfully deleted
I20260809 22:42:08.291714 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.261075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.28.243.65:34975)
0809 22:42:08.261265 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.261272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.265725 (+ 4453us) server_negotiation.cc:407] Connection header received
0809 22:42:08.266188 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.266196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.266286 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.266371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:08.270131 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.271470 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.274823 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.275638 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.275763 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.283369 (+ 7606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:08.283387 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:08.283407 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:08.283470 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:08.288285 (+ 4815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.289281 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.289291 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.289295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.289493 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.289912 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.289918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.289921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.290479 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:08.290827 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.291016 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.291264 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:42:08.292538 16914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:08.389178 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 16786
I20260809 22:42:08.419399 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 16928
Aug 09 22:42:08 dist-test-slave-rgb1 krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:42:08.440047 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.420242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.28.243.66:38645)
0809 22:42:08.420457 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.420464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.420489 (+ 25us) server_negotiation.cc:407] Connection header received
0809 22:42:08.420647 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.420653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.420768 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.420901 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:08.422359 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.423646 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.425444 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.426248 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.426411 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.432107 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:08.432125 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:08.432142 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:08.432187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:08.435937 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.437080 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.437089 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.437091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.437254 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.437766 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.437770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.437772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.438277 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:08.438570 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.439020 (+ 450us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.439509 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 22:42:08.440968 17054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:08.563481 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 16928
I20260809 22:42:08.584738 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 17068
Aug 09 22:42:08 dist-test-slave-rgb1 krb5kdc[16674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:42:08.601998 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:08.585646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.28.243.67:37711)
0809 22:42:08.585858 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:08.585864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:08.585882 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:42:08.585944 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:08.585948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:08.586015 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:08.586131 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:08.587646 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.588553 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.590019 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:08.590425 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:08.590534 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:08.595351 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:08.595367 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:08.595384 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:08.595428 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:08.598745 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.599516 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.599524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.599526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.599734 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:08.600127 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:08.600131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:08.600137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:08.600540 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:08.600939 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:08.601427 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 22:42:08.601635 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:42:08.602770 17193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:08.697072 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 17068
I20260809 22:42:08.718426 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 16687
I20260809 22:42:08.719621 16752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:08.833210 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 16687
2026-08-09T22:42:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4813 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T22:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:08Z Disabled control of system clock
I20260809 22:42:08.934854 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38757
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:38757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 22:42:09.319432 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:09.319746 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:09.319839 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:09.327777 17253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:09.328002 17253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:09.328099 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:09.328173 17253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:09.328222 17253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:09.338493 17253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:38757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:09.339884 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:09.341861 17253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:09.355577 17259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:09.355528 17260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:09.355489 17262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:09.357120 17253 server_base.cc:1034] running on GCE node
I20260809 22:42:09.358019 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:09.359635 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:09.360915 17253 hybrid_clock.cc:648] HybridClock initialized: now 1786315329360869 us; error 75 us; skew 500 ppm
I20260809 22:42:09.361351 17253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:09.364354 17253 webserver.cc:460] Webserver started at http://127.28.243.126:36227/ using document root <none> and password file <none>
I20260809 22:42:09.365267 17253 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:09.365434 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:09.365826 17253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:09.369359 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "17c54a8d72b641789012206cbfcbb3aa"
format_stamp: "Formatted at 2026-08-09 22:42:09 on dist-test-slave-rgb1"
server_key: "321d83d3e520ff8eb49264660075d9a3"
server_key_iv: "c6e6ee7e9554ed096578fd1feba4f5c0"
server_key_version: "encryptionkey@0"
I20260809 22:42:09.370434 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "17c54a8d72b641789012206cbfcbb3aa"
format_stamp: "Formatted at 2026-08-09 22:42:09 on dist-test-slave-rgb1"
server_key: "321d83d3e520ff8eb49264660075d9a3"
server_key_iv: "c6e6ee7e9554ed096578fd1feba4f5c0"
server_key_version: "encryptionkey@0"
I20260809 22:42:09.376650 17253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 22:42:09.380712 17269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:09.382208 17253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 22:42:09.382426 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "17c54a8d72b641789012206cbfcbb3aa"
format_stamp: "Formatted at 2026-08-09 22:42:09 on dist-test-slave-rgb1"
server_key: "321d83d3e520ff8eb49264660075d9a3"
server_key_iv: "c6e6ee7e9554ed096578fd1feba4f5c0"
server_key_version: "encryptionkey@0"
I20260809 22:42:09.382669 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:09.398653 17253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:09.400007 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:09.400378 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:09.425985 17253 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:38757
I20260809 22:42:09.426013 17323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:38757 every 8 connection(s)
I20260809 22:42:09.427649 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:09.430613 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 17253
I20260809 22:42:09.430815 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:09.431397 29645 external_mini_cluster.cc:1428] Setting key 1837a9f9cf0ad5a49eb84e4c2a5ff389
I20260809 22:42:09.432299 17324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:09.449215 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:09.433673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35468 (local address 127.28.243.126:38757)
0809 22:42:09.435029 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:09.435041 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:09.435265 (+ 224us) server_negotiation.cc:407] Connection header received
0809 22:42:09.438031 (+ 2766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:09.438056 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:09.438596 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:09.438982 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:09.439743 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:09.441294 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:09.443292 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:09.444185 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:09.444379 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:09.445931 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:09.445940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:09.445956 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:09.446720 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:09.446810 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:09.446836 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:09.446946 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:09.447297 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 22:42:09.447893 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":477,"threads_started":1}
I20260809 22:42:09.449288 17324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa: Bootstrap starting.
I20260809 22:42:09.453425 17324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:09.454516 17324 log.cc:824] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:09.458029 17324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa: No bootstrap required, opened a new log
I20260809 22:42:09.466612 17324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c54a8d72b641789012206cbfcbb3aa" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38757 } }
I20260809 22:42:09.467088 17324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:09.467232 17324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c54a8d72b641789012206cbfcbb3aa, State: Initialized, Role: FOLLOWER
I20260809 22:42:09.467851 17324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c54a8d72b641789012206cbfcbb3aa" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38757 } }
I20260809 22:42:09.468130 17324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:09.468302 17324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:09.468492 17324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:09.471397 17324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c54a8d72b641789012206cbfcbb3aa" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38757 } }
I20260809 22:42:09.472035 17324 leader_election.cc:302] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c54a8d72b641789012206cbfcbb3aa; no voters:
I20260809 22:42:09.472857 17324 leader_election.cc:288] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:09.473112 17329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:09.474329 17329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [term 1 LEADER]: Becoming Leader. State: Replica: 17c54a8d72b641789012206cbfcbb3aa, State: Running, Role: LEADER
I20260809 22:42:09.475359 17324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:09.475198 17329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c54a8d72b641789012206cbfcbb3aa" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38757 } }
I20260809 22:42:09.479118 17330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c54a8d72b641789012206cbfcbb3aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c54a8d72b641789012206cbfcbb3aa" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38757 } } }
I20260809 22:42:09.479435 17331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c54a8d72b641789012206cbfcbb3aa. Latest consensus state: current_term: 1 leader_uuid: "17c54a8d72b641789012206cbfcbb3aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c54a8d72b641789012206cbfcbb3aa" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 38757 } } }
I20260809 22:42:09.479700 17330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:09.479765 17331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:09.482232 17339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:09.488328 17339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:09.499694 17339 catalog_manager.cc:1347] Generated new cluster ID: cb045306842949b999dd86caebe3e384
I20260809 22:42:09.499876 17339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:09.523722 17339 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:09.525718 17339 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:09.543087 17339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa: Generated new TSK 0
I20260809 22:42:09.544265 17339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:09.553843 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38757
--builtin_ntp_servers=127.28.243.84:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 22:42:09.949550 17348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:09.949914 17348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:09.950022 17348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:09.957680 17348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:09.957849 17348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:09.958039 17348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:42:09.969054 17348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:09.970567 17348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:09.972477 17348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:09.987890 17355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:09.988458 17354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:09.989040 17357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:09.989073 17348 server_base.cc:1034] running on GCE node
I20260809 22:42:09.989938 17348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:09.991118 17348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:09.992348 17348 hybrid_clock.cc:648] HybridClock initialized: now 1786315329992275 us; error 77 us; skew 500 ppm
I20260809 22:42:09.992764 17348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:09.995437 17348 webserver.cc:460] Webserver started at http://127.28.243.65:36979/ using document root <none> and password file <none>
I20260809 22:42:09.996134 17348 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:09.996227 17348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:09.996495 17348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:09.999594 17348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04"
format_stamp: "Formatted at 2026-08-09 22:42:09 on dist-test-slave-rgb1"
server_key: "4c98a0fcc745002f1ed06f839b11e170"
server_key_iv: "5da7520ffd1e3b485e0dfbf36ed8be96"
server_key_version: "encryptionkey@0"
I20260809 22:42:10.000360 17348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04"
format_stamp: "Formatted at 2026-08-09 22:42:09 on dist-test-slave-rgb1"
server_key: "4c98a0fcc745002f1ed06f839b11e170"
server_key_iv: "5da7520ffd1e3b485e0dfbf36ed8be96"
server_key_version: "encryptionkey@0"
I20260809 22:42:10.008706 17348 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260809 22:42:10.014261 17364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:10.016072 17348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260809 22:42:10.016323 17348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04"
format_stamp: "Formatted at 2026-08-09 22:42:09 on dist-test-slave-rgb1"
server_key: "4c98a0fcc745002f1ed06f839b11e170"
server_key_iv: "5da7520ffd1e3b485e0dfbf36ed8be96"
server_key_version: "encryptionkey@0"
I20260809 22:42:10.016621 17348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:10.040256 17348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:10.041322 17348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:10.041707 17348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:10.043004 17348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:10.044768 17348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:10.044906 17348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:10.045073 17348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:10.045184 17348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:10.087744 17348 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:37065
I20260809 22:42:10.087879 17479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:37065 every 8 connection(s)
I20260809 22:42:10.089490 17348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:42:10.093817 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 17348
I20260809 22:42:10.094054 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:42:10.094712 29645 external_mini_cluster.cc:1428] Setting key 66b28ad6ed6f2a0534fa45a9b13bcb5a
I20260809 22:42:10.098480 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38757
--builtin_ntp_servers=127.28.243.84:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 22:42:10.105747 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:10.092642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:50749 (local address 127.28.243.126:38757)
0809 22:42:10.093330 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:10.093335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:10.094342 (+ 1007us) server_negotiation.cc:407] Connection header received
0809 22:42:10.096179 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:10.096185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:10.096278 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:10.096440 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:10.097737 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.099024 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.101465 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.102282 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.102540 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:10.104466 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:10.104477 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:10.104485 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:10.104517 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:10.104598 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:10.104621 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:10.104731 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:10.105073 (+ 342us) server_negotiation.cc:299] Negotiation successful
0809 22:42:10.105210 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":419,"threads_started":1}
I20260809 22:42:10.106804 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:10.093043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38757 (local address 127.28.243.65:50749)
0809 22:42:10.094141 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:10.094188 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:10.095901 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:10.096815 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:10.096824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:10.096914 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:10.097532 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:10.097551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.099237 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.099242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:10.101255 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:10.101271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.102645 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.102652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:10.102893 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:10.103919 (+ 1026us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:10.103923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:10.104057 (+ 134us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:10.104174 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:10.104792 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:10.104804 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:10.104811 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 22:42:10.105072 (+ 261us) client_negotiation.cc:240] Negotiation successful
0809 22:42:10.105464 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":389,"threads_started":1}
I20260809 22:42:10.109084 17480 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38757
I20260809 22:42:10.109463 17480 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:10.110334 17480 heartbeater.cc:511] Master 127.28.243.126:38757 requested a full tablet report, sending...
I20260809 22:42:10.114122 17289 ts_manager.cc:194] Registered new tserver with Master: eae3e9462bfa4dd68d5b8f77d6c46e04 (127.28.243.65:37065)
I20260809 22:42:10.116711 17289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.65:50749
W20260809 22:42:10.482156 17484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:10.482566 17484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:10.482751 17484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:10.491293 17484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:10.491463 17484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:10.491714 17484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:42:10.502327 17484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:10.503731 17484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:10.505594 17484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:10.520792 17490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:10.520816 17492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:10.521250 17489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:10.521827 17484 server_base.cc:1034] running on GCE node
I20260809 22:42:10.522598 17484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:10.523805 17484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:10.525251 17484 hybrid_clock.cc:648] HybridClock initialized: now 1786315330525143 us; error 60 us; skew 500 ppm
I20260809 22:42:10.525911 17484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:10.528468 17484 webserver.cc:460] Webserver started at http://127.28.243.66:36921/ using document root <none> and password file <none>
I20260809 22:42:10.529407 17484 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:10.529549 17484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:10.529886 17484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:10.533036 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "ead2a98ac4d14f51847f732a55638e9e"
format_stamp: "Formatted at 2026-08-09 22:42:10 on dist-test-slave-rgb1"
server_key: "67e3e91b7d9c15d118ecc529f42e8b71"
server_key_iv: "6d27723c741c116f39bb2c01ddcedab7"
server_key_version: "encryptionkey@0"
I20260809 22:42:10.533907 17484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "ead2a98ac4d14f51847f732a55638e9e"
format_stamp: "Formatted at 2026-08-09 22:42:10 on dist-test-slave-rgb1"
server_key: "67e3e91b7d9c15d118ecc529f42e8b71"
server_key_iv: "6d27723c741c116f39bb2c01ddcedab7"
server_key_version: "encryptionkey@0"
I20260809 22:42:10.539623 17484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:42:10.543390 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:10.545011 17484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 22:42:10.545202 17484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "ead2a98ac4d14f51847f732a55638e9e"
format_stamp: "Formatted at 2026-08-09 22:42:10 on dist-test-slave-rgb1"
server_key: "67e3e91b7d9c15d118ecc529f42e8b71"
server_key_iv: "6d27723c741c116f39bb2c01ddcedab7"
server_key_version: "encryptionkey@0"
I20260809 22:42:10.545439 17484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:10.566151 17484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:10.567184 17484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:10.567585 17484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:10.569172 17484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:10.570962 17484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:10.571120 17484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:10.571267 17484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:10.571416 17484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:10.612336 17484 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:45231
I20260809 22:42:10.612402 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:45231 every 8 connection(s)
I20260809 22:42:10.613912 17484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:42:10.621280 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 17484
I20260809 22:42:10.621524 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:42:10.622212 29645 external_mini_cluster.cc:1428] Setting key 4dc9c33157b63ffb32c6ef03de04a15b
I20260809 22:42:10.626189 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38757
--builtin_ntp_servers=127.28.243.84:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 22:42:10.629990 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:10.616921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:60137 (local address 127.28.243.126:38757)
0809 22:42:10.617482 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:10.617486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:10.618824 (+ 1338us) server_negotiation.cc:407] Connection header received
0809 22:42:10.621022 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:10.621027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:10.621118 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:10.621230 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:10.622944 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.623983 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.626254 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.627047 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.627218 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:10.628842 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:10.628868 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:10.628876 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:10.628904 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:10.628961 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:10.628982 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:10.629082 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:10.629400 (+ 318us) server_negotiation.cc:299] Negotiation successful
0809 22:42:10.629538 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":317,"threads_started":1}
I20260809 22:42:10.631054 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:10.617540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38757 (local address 127.28.243.66:60137)
0809 22:42:10.618633 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:10.618673 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:10.620516 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:10.621552 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:10.621567 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:10.621697 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:10.622640 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:10.622671 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.624363 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.624370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:10.626061 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:10.626076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:10.627308 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:10.627314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:10.627577 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:10.628375 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:10.628378 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:10.628422 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:10.628525 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:10.629122 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:10.629132 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:10.629139 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 22:42:10.629334 (+ 195us) client_negotiation.cc:240] Negotiation successful
0809 22:42:10.629720 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":376,"threads_started":1}
I20260809 22:42:10.632916 17616 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38757
I20260809 22:42:10.633277 17616 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:10.634287 17616 heartbeater.cc:511] Master 127.28.243.126:38757 requested a full tablet report, sending...
I20260809 22:42:10.636103 17289 ts_manager.cc:194] Registered new tserver with Master: ead2a98ac4d14f51847f732a55638e9e (127.28.243.66:45231)
I20260809 22:42:10.637058 17289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.66:60137
W20260809 22:42:11.002208 17620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:11.002537 17620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:11.002722 17620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:11.010725 17620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:11.010965 17620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:11.011186 17620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:42:11.022493 17620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:38757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:11.024629 17620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:11.026813 17620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:11.042065 17626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:11.042101 17625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:11.042066 17628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:11.043563 17620 server_base.cc:1034] running on GCE node
I20260809 22:42:11.044353 17620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:11.045621 17620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:11.046871 17620 hybrid_clock.cc:648] HybridClock initialized: now 1786315331046801 us; error 56 us; skew 500 ppm
I20260809 22:42:11.047232 17620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:11.056711 17620 webserver.cc:460] Webserver started at http://127.28.243.67:45631/ using document root <none> and password file <none>
I20260809 22:42:11.057596 17620 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:11.057713 17620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:11.058053 17620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:11.061450 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "7b462d31bd8e45c7a66788bbf9b06852"
format_stamp: "Formatted at 2026-08-09 22:42:11 on dist-test-slave-rgb1"
server_key: "c6413af9506bc58fca421911b0056596"
server_key_iv: "3e852ab289cd8ffdbcd6d8067422e7f9"
server_key_version: "encryptionkey@0"
I20260809 22:42:11.062397 17620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "7b462d31bd8e45c7a66788bbf9b06852"
format_stamp: "Formatted at 2026-08-09 22:42:11 on dist-test-slave-rgb1"
server_key: "c6413af9506bc58fca421911b0056596"
server_key_iv: "3e852ab289cd8ffdbcd6d8067422e7f9"
server_key_version: "encryptionkey@0"
I20260809 22:42:11.068781 17620 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260809 22:42:11.073518 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:11.075366 17620 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 22:42:11.075537 17620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "7b462d31bd8e45c7a66788bbf9b06852"
format_stamp: "Formatted at 2026-08-09 22:42:11 on dist-test-slave-rgb1"
server_key: "c6413af9506bc58fca421911b0056596"
server_key_iv: "3e852ab289cd8ffdbcd6d8067422e7f9"
server_key_version: "encryptionkey@0"
I20260809 22:42:11.075917 17620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:11.099088 17620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:11.100036 17620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:11.100395 17620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:11.101557 17620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:11.103293 17620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:11.103395 17620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:11.103554 17620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:11.103644 17620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:11.121043 17480 heartbeater.cc:503] Master 127.28.243.126:38757 was elected leader, sending a full tablet report...
I20260809 22:42:11.150616 17620 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:37973
I20260809 22:42:11.150722 17753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:37973 every 8 connection(s)
I20260809 22:42:11.152037 17620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:42:11.155613 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 17620
I20260809 22:42:11.155917 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:42:11.156997 29645 external_mini_cluster.cc:1428] Setting key ec6b10d37a41efa5e068333b9a2f4fbc
I20260809 22:42:11.168974 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.154993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:35927 (local address 127.28.243.126:38757)
0809 22:42:11.155456 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.155461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.158011 (+ 2550us) server_negotiation.cc:407] Connection header received
0809 22:42:11.159950 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.159956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.160049 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.160190 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:11.161397 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.162453 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.165466 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.165971 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.166139 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.167866 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:11.167871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:11.167875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:11.167898 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:11.167944 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:11.167957 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:11.168074 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.168337 (+ 263us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.168464 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":254,"threads_started":1}
I20260809 22:42:11.170090 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.156380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:38757 (local address 127.28.243.67:35927)
0809 22:42:11.157825 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.157872 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:11.159660 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:11.160443 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:11.160451 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:11.160545 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:11.161219 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.161237 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.162671 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.162676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.165248 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.165265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.166249 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.166257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.166586 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.167474 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:11.167477 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:11.167524 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:11.167619 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:11.168073 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:11.168087 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:11.168098 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 22:42:11.168308 (+ 210us) client_negotiation.cc:240] Negotiation successful
0809 22:42:11.168862 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":403,"threads_started":1}
I20260809 22:42:11.171918 17754 heartbeater.cc:347] Connected to a master server at 127.28.243.126:38757
I20260809 22:42:11.172336 17754 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:11.173416 17754 heartbeater.cc:511] Master 127.28.243.126:38757 requested a full tablet report, sending...
I20260809 22:42:11.175336 17287 ts_manager.cc:194] Registered new tserver with Master: 7b462d31bd8e45c7a66788bbf9b06852 (127.28.243.67:37973)
I20260809 22:42:11.176640 17287 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.67:35927
I20260809 22:42:11.186262 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:42:11.198710 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.191708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.28.243.126:38757)
0809 22:42:11.191953 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.191958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.192653 (+ 695us) server_negotiation.cc:407] Connection header received
0809 22:42:11.192864 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.192868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.192942 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.193187 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:11.193704 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.194496 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.195949 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.196416 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.196528 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.197827 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:11.197832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:11.197835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:11.197855 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:11.197887 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:11.197899 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:11.198055 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.198285 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.198379 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 22:42:11.204120 17287 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:42:11.212258 17287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:42:11.243253 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.229222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:45231 (local address 127.0.0.1:32836)
0809 22:42:11.229976 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.230017 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:11.230202 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:11.231824 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:11.231839 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:11.231883 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:11.232448 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.232468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.235201 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.235212 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.238447 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.238477 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.241642 (+ 3165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.241653 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.242055 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.242148 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 22:42:11.242361 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 22:42:11.242673 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":303,"threads_started":1}
I20260809 22:42:11.244752 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.229399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.28.243.66:45231)
0809 22:42:11.230780 (+ 1381us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.230790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.230824 (+ 34us) server_negotiation.cc:407] Connection header received
0809 22:42:11.230941 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.230949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.231237 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.231496 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:11.233011 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.234954 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.238817 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.240883 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.241052 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.241412 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.243916 (+ 2504us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.244206 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":1141,"threads_started":1}
I20260809 22:42:11.249419 17549 tablet_service.cc:1467] Processing CreateTablet for tablet 63b548a8606b48b391b85f651825c00c (DEFAULT_TABLE table=test-table [id=cb0d46c5578d4f51b02edde729e57258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:11.251120 17549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b548a8606b48b391b85f651825c00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:11.253957 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.231254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40840)
0809 22:42:11.244539 (+ 13285us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.244583 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:11.244825 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:11.246005 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:11.246008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:11.246030 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:11.246537 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.246553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.249099 (+ 2546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.249105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.251299 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.251325 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.252893 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.252901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.253132 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.253175 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 22:42:11.253404 (+ 229us) client_negotiation.cc:240] Negotiation successful
0809 22:42:11.253568 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13150,"thread_start_us":12993,"threads_started":1}
I20260809 22:42:11.254035 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.231553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.28.243.65:37065)
0809 22:42:11.243792 (+ 12239us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.243805 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.244696 (+ 891us) server_negotiation.cc:407] Connection header received
0809 22:42:11.245066 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.245076 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.245406 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.245733 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:11.246763 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.248892 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.251536 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.252646 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.252854 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.253164 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.253447 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.253643 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11991,"thread_start_us":12267,"threads_started":1}
I20260809 22:42:11.258234 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.231929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:37973 (local address 127.0.0.1:60658)
0809 22:42:11.246448 (+ 14519us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.246481 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:11.246645 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:11.248099 (+ 1454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:11.248107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:11.248141 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:11.248384 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.248395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.250576 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.250582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.252917 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.252952 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.257370 (+ 4418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.257379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.257590 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.257624 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 22:42:11.257733 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 22:42:11.257913 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14303,"thread_start_us":16100,"threads_started":1}
I20260809 22:42:11.258476 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.232968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.28.243.67:37973)
0809 22:42:11.241194 (+ 8226us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.241207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.246993 (+ 5786us) server_negotiation.cc:407] Connection header received
0809 22:42:11.247170 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.247182 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.247517 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.247771 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:11.248634 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.250382 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.254759 (+ 4377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.256794 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.256959 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.257245 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.257820 (+ 575us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.258020 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7967,"thread_start_us":7758,"threads_started":1}
I20260809 22:42:11.258596 17414 tablet_service.cc:1467] Processing CreateTablet for tablet 63b548a8606b48b391b85f651825c00c (DEFAULT_TABLE table=test-table [id=cb0d46c5578d4f51b02edde729e57258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:11.260150 17414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b548a8606b48b391b85f651825c00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:11.266750 17773 tablet_bootstrap.cc:492] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Bootstrap starting.
I20260809 22:42:11.269699 17684 tablet_service.cc:1467] Processing CreateTablet for tablet 63b548a8606b48b391b85f651825c00c (DEFAULT_TABLE table=test-table [id=cb0d46c5578d4f51b02edde729e57258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:11.270813 17773 tablet_bootstrap.cc:654] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:11.271287 17684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63b548a8606b48b391b85f651825c00c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:11.272207 17773 log.cc:824] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:11.275466 17775 tablet_bootstrap.cc:492] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: Bootstrap starting.
I20260809 22:42:11.275835 17773 tablet_bootstrap.cc:492] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: No bootstrap required, opened a new log
I20260809 22:42:11.276157 17773 ts_tablet_manager.cc:1397] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260809 22:42:11.280579 17775 tablet_bootstrap.cc:654] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:11.281942 17775 log.cc:824] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:11.285385 17775 tablet_bootstrap.cc:492] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: No bootstrap required, opened a new log
I20260809 22:42:11.285701 17775 ts_tablet_manager.cc:1397] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260809 22:42:11.285542 17773 raft_consensus.cc:348] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.285986 17773 raft_consensus.cc:374] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:11.286175 17773 raft_consensus.cc:729] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead2a98ac4d14f51847f732a55638e9e, State: Initialized, Role: FOLLOWER
I20260809 22:42:11.286850 17773 consensus_queue.cc:260] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.293713 17777 tablet_bootstrap.cc:492] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Bootstrap starting.
I20260809 22:42:11.295416 17616 heartbeater.cc:503] Master 127.28.243.126:38757 was elected leader, sending a full tablet report...
I20260809 22:42:11.296231 17773 ts_tablet_manager.cc:1428] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260809 22:42:11.298406 17777 tablet_bootstrap.cc:654] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:11.298539 17775 raft_consensus.cc:348] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.299034 17775 raft_consensus.cc:374] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:11.299152 17775 raft_consensus.cc:729] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae3e9462bfa4dd68d5b8f77d6c46e04, State: Initialized, Role: FOLLOWER
I20260809 22:42:11.299978 17775 consensus_queue.cc:260] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.300436 17777 log.cc:824] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:11.302556 17775 ts_tablet_manager.cc:1428] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260809 22:42:11.304313 17777 tablet_bootstrap.cc:492] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: No bootstrap required, opened a new log
I20260809 22:42:11.304670 17777 ts_tablet_manager.cc:1397] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260809 22:42:11.312134 17777 raft_consensus.cc:348] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.312601 17777 raft_consensus.cc:374] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:11.312726 17777 raft_consensus.cc:729] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b462d31bd8e45c7a66788bbf9b06852, State: Initialized, Role: FOLLOWER
I20260809 22:42:11.313427 17777 consensus_queue.cc:260] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.314568 17754 heartbeater.cc:503] Master 127.28.243.126:38757 was elected leader, sending a full tablet report...
I20260809 22:42:11.315062 17777 ts_tablet_manager.cc:1428] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
W20260809 22:42:11.342697 17481 tablet.cc:2369] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:11.361001 17780 raft_consensus.cc:482] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:42:11.361356 17780 raft_consensus.cc:504] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.363627 17780 leader_election.cc:288] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ead2a98ac4d14f51847f732a55638e9e (127.28.243.66:45231), 7b462d31bd8e45c7a66788bbf9b06852 (127.28.243.67:37973)
W20260809 22:42:11.366642 17617 tablet.cc:2369] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:11.374327 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.364448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:33545 (local address 127.28.243.67:37973)
0809 22:42:11.364718 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.364727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.365370 (+ 643us) server_negotiation.cc:407] Connection header received
0809 22:42:11.365587 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.365591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.365650 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.365756 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:11.366509 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.367917 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.371698 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.373032 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.373228 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.373302 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.373833 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.373969 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:42:11.374593 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.364116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:37973 (local address 127.28.243.65:33545)
0809 22:42:11.365254 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.365269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:11.365436 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:11.365776 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:11.365780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:11.365820 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:11.366281 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.366334 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.368408 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.368416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.371380 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.371401 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.373358 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.373367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.373611 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.373635 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 22:42:11.373740 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 22:42:11.373993 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":371,"thread_start_us":494,"threads_started":1}
I20260809 22:42:11.375098 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.364092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:36605 (local address 127.28.243.66:45231)
0809 22:42:11.364355 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.364365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.365144 (+ 779us) server_negotiation.cc:407] Connection header received
0809 22:42:11.365400 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.365411 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.365495 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.365617 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:11.366290 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.367934 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.371698 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.372993 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.373227 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.373302 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.374680 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.374817 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 22:42:11.375419 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.363887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:45231 (local address 127.28.243.65:36605)
0809 22:42:11.364997 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.365020 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:11.365234 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:11.365644 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:11.365649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:11.365675 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:11.366118 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.366132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.368277 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.368282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.371380 (+ 3098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:11.371401 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.373377 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.373383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:11.373576 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.373615 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 22:42:11.373740 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 22:42:11.373989 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":314,"thread_start_us":1038,"threads_started":1}
I20260809 22:42:11.376035 17705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b548a8606b48b391b85f651825c00c" candidate_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b462d31bd8e45c7a66788bbf9b06852" is_pre_election: true
I20260809 22:42:11.376456 17569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b548a8606b48b391b85f651825c00c" candidate_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead2a98ac4d14f51847f732a55638e9e" is_pre_election: true
I20260809 22:42:11.376720 17705 raft_consensus.cc:2454] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae3e9462bfa4dd68d5b8f77d6c46e04 in term 0.
I20260809 22:42:11.377064 17569 raft_consensus.cc:2454] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eae3e9462bfa4dd68d5b8f77d6c46e04 in term 0.
I20260809 22:42:11.377574 17367 leader_election.cc:302] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b462d31bd8e45c7a66788bbf9b06852, eae3e9462bfa4dd68d5b8f77d6c46e04; no voters:
I20260809 22:42:11.378027 17780 raft_consensus.cc:2781] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:42:11.378196 17780 raft_consensus.cc:482] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:42:11.378334 17780 raft_consensus.cc:3037] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:11.381112 17780 raft_consensus.cc:504] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.381736 17780 leader_election.cc:288] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [CANDIDATE]: Term 1 election: Requested vote from peers ead2a98ac4d14f51847f732a55638e9e (127.28.243.66:45231), 7b462d31bd8e45c7a66788bbf9b06852 (127.28.243.67:37973)
I20260809 22:42:11.382385 17705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b548a8606b48b391b85f651825c00c" candidate_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b462d31bd8e45c7a66788bbf9b06852"
I20260809 22:42:11.382406 17569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63b548a8606b48b391b85f651825c00c" candidate_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead2a98ac4d14f51847f732a55638e9e"
I20260809 22:42:11.382649 17705 raft_consensus.cc:3037] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:11.382649 17569 raft_consensus.cc:3037] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:11.385955 17705 raft_consensus.cc:2454] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae3e9462bfa4dd68d5b8f77d6c46e04 in term 1.
I20260809 22:42:11.385991 17569 raft_consensus.cc:2454] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eae3e9462bfa4dd68d5b8f77d6c46e04 in term 1.
I20260809 22:42:11.386472 17366 leader_election.cc:302] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ead2a98ac4d14f51847f732a55638e9e, eae3e9462bfa4dd68d5b8f77d6c46e04; no voters:
I20260809 22:42:11.386862 17780 raft_consensus.cc:2781] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:11.387416 17780 raft_consensus.cc:686] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 1 LEADER]: Becoming Leader. State: Replica: eae3e9462bfa4dd68d5b8f77d6c46e04, State: Running, Role: LEADER
I20260809 22:42:11.388005 17780 consensus_queue.cc:237] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } }
I20260809 22:42:11.393640 17289 catalog_manager.cc:5495] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 reported cstate change: term changed from 0 to 1, leader changed from <none> to eae3e9462bfa4dd68d5b8f77d6c46e04 (127.28.243.65). New cstate: current_term: 1 leader_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eae3e9462bfa4dd68d5b8f77d6c46e04" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 37065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 } health_report { overall_health: UNKNOWN } } }
W20260809 22:42:11.404037 17755 tablet.cc:2369] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:11.425124 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.418495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.28.243.65:37065)
0809 22:42:11.418737 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.418746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.418767 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:42:11.419064 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.419075 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.419188 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.419358 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:42:11.419773 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.420517 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.422085 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.422584 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.422728 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.423319 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:42:11.424202 (+ 883us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:42:11.424463 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.424651 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.424703 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 22:42:11.435896 17705 raft_consensus.cc:1261] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 1 FOLLOWER]: Refusing update from remote peer eae3e9462bfa4dd68d5b8f77d6c46e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:11.435854 17569 raft_consensus.cc:1261] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 1 FOLLOWER]: Refusing update from remote peer eae3e9462bfa4dd68d5b8f77d6c46e04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:11.437090 17785 consensus_queue.cc:1035] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ead2a98ac4d14f51847f732a55638e9e" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 45231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:11.437597 17780 consensus_queue.cc:1035] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b462d31bd8e45c7a66788bbf9b06852" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 37973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:11.451359 17788 mvcc.cc:205] Tried to move back new op lower bound from 7316747597546090496 to 7316747597370957824. Current Snapshot: MvccSnapshot[applied={T|T < 7316747597546090496}]
I20260809 22:42:11.464671 17790 mvcc.cc:205] Tried to move back new op lower bound from 7316747597546090496 to 7316747597370957824. Current Snapshot: MvccSnapshot[applied={T|T < 7316747597546090496}]
I20260809 22:42:11.466686 17287 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:11.467007 17287 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:11.471025 17287 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17c54a8d72b641789012206cbfcbb3aa: Sending DeleteTablet for 3 replicas of tablet 63b548a8606b48b391b85f651825c00c
I20260809 22:42:11.472687 17414 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b548a8606b48b391b85f651825c00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:11 UTC) from {username='slave'} at 127.0.0.1:40840
I20260809 22:42:11.472676 17684 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b548a8606b48b391b85f651825c00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:11 UTC) from {username='slave'} at 127.0.0.1:60658
I20260809 22:42:11.473520 17549 tablet_service.cc:1514] Processing DeleteTablet for tablet 63b548a8606b48b391b85f651825c00c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:11 UTC) from {username='slave'} at 127.0.0.1:32836
I20260809 22:42:11.474404 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 17348
I20260809 22:42:11.495814 17799 tablet_replica.cc:330] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: stopping tablet replica
I20260809 22:42:11.496371 17799 raft_consensus.cc:2229] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:11.498965 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.475967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.28.243.65:37065)
0809 22:42:11.476215 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.476225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.487110 (+ 10885us) server_negotiation.cc:407] Connection header received
0809 22:42:11.487850 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.487858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.488010 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.488118 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:11.488995 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.490489 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.493064 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.494147 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.494266 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.497057 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:11.497074 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:11.497093 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:11.497958 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:11.498023 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:11.498052 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:11.498164 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.498407 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.498544 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 22:42:11.500463 17475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:11.500465 17799 raft_consensus.cc:2258] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:11.501536 17798 tablet_replica.cc:330] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: stopping tablet replica
I20260809 22:42:11.502391 17798 raft_consensus.cc:2229] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:11.503118 17798 raft_consensus.cc:2258] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:11.503737 17799 ts_tablet_manager.cc:1905] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:11.506136 17798 ts_tablet_manager.cc:1905] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:11.513216 17799 ts_tablet_manager.cc:1918] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:11.513441 17799 log.cc:1196] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/63b548a8606b48b391b85f651825c00c
I20260809 22:42:11.513944 17799 ts_tablet_manager.cc:1939] T 63b548a8606b48b391b85f651825c00c P ead2a98ac4d14f51847f732a55638e9e: Deleting consensus metadata
I20260809 22:42:11.516203 17272 catalog_manager.cc:4841] TS ead2a98ac4d14f51847f732a55638e9e (127.28.243.66:45231): tablet 63b548a8606b48b391b85f651825c00c (table test-table [id=cb0d46c5578d4f51b02edde729e57258]) successfully deleted
I20260809 22:42:11.516497 17798 ts_tablet_manager.cc:1918] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:11.516799 17798 log.cc:1196] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/63b548a8606b48b391b85f651825c00c
I20260809 22:42:11.517405 17798 ts_tablet_manager.cc:1939] T 63b548a8606b48b391b85f651825c00c P 7b462d31bd8e45c7a66788bbf9b06852: Deleting consensus metadata
I20260809 22:42:11.519562 17273 catalog_manager.cc:4841] TS 7b462d31bd8e45c7a66788bbf9b06852 (127.28.243.67:37973): tablet 63b548a8606b48b391b85f651825c00c (table test-table [id=cb0d46c5578d4f51b02edde729e57258]) successfully deleted
I20260809 22:42:11.615801 17797 tablet_replica.cc:330] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04: stopping tablet replica
I20260809 22:42:11.616330 17797 raft_consensus.cc:2229] T 63b548a8606b48b391b85f651825c00c P eae3e9462bfa4dd68d5b8f77d6c46e04 [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:42:11.616869 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 17348
I20260809 22:42:11.645246 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 17484
W20260809 22:42:11.645341 17274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.243.65:37065 (error 108)
I20260809 22:42:11.648152 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.647634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40868)
0809 22:42:11.647876 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.647990 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 22:42:11.648459 17274 catalog_manager.cc:4584] TS eae3e9462bfa4dd68d5b8f77d6c46e04 (127.28.243.65:37065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63b548a8606b48b391b85f651825c00c: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
I20260809 22:42:11.654915 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.646212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.28.243.66:45231)
0809 22:42:11.646943 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.646948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.646965 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:42:11.647019 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.647023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.647097 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.647229 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:11.647720 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.649056 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.650758 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.651549 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.651688 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.653178 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:11.653190 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:11.653207 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:11.654038 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:11.654101 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:11.654126 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:11.654237 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.654407 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.654532 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594}
I20260809 22:42:11.655745 17611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:11.702651 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.702138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40878)
0809 22:42:11.702374 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.702500 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 22:42:11.703943 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.703532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40890)
0809 22:42:11.703703 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.703778 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 22:42:11.753155 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 17484
I20260809 22:42:11.764474 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.764004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40898)
0809 22:42:11.764230 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.764321 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 22:42:11.765610 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.765258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40902)
0809 22:42:11.765408 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.765485 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 22:42:11.772656 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 17620
I20260809 22:42:11.780500 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.773628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.28.243.67:37973)
0809 22:42:11.773939 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:11.773946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:11.773969 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:42:11.774051 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:11.774055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:11.774142 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:11.774261 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:11.774672 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.775507 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.777067 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:11.777483 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:11.777628 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:11.778877 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:11.778887 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:11.778924 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:11.779533 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:11.779600 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:11.779621 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:11.779748 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:11.779890 (+ 142us) server_negotiation.cc:299] Negotiation successful
0809 22:42:11.780110 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260809 22:42:11.781376 17749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:11.846788 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.846299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40912)
0809 22:42:11.846537 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.846638 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260809 22:42:11.847903 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:11.847448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:37065 (local address 127.0.0.1:40920)
0809 22:42:11.847664 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:11.847749 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.65:37065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260809 22:42:11.873418 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 17620
I20260809 22:42:11.893465 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 17253
I20260809 22:42:11.894837 17319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:12.009665 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 17253
2026-08-09T22:42:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:12 dist-test-slave-rgb1 krb5kdc[17810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:12 dist-test-slave-rgb1 krb5kdc[17810](info): set up 2 sockets
Aug 09 22:42:12 dist-test-slave-rgb1 krb5kdc[17810](info): commencing operation
krb5kdc: starting...
W20260809 22:42:13.233460 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.003s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:13 dist-test-slave-rgb1 krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:13.588832 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34055
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:34055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:13.975941 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:13.976333 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:13.976503 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:13.984141 17823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:13.984324 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:13.984438 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:13.984549 17823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:13.984669 17823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:13.995185 17823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:34055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:13.996744 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:13.998183 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:14.012115 17829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:14.012149 17830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:14.012151 17832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:14.013037 17823 server_base.cc:1034] running on GCE node
I20260809 22:42:14.013948 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:14.015488 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:14.016742 17823 hybrid_clock.cc:648] HybridClock initialized: now 1786315334016687 us; error 30 us; skew 500 ppm
I20260809 22:42:14.017195 17823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:14 dist-test-slave-rgb1 krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315334, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:14.020535 17823 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:42:14.022352 17823 webserver.cc:460] Webserver started at http://127.28.243.126:33961/ using document root <none> and password file <none>
I20260809 22:42:14.023085 17823 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:14.023193 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:14.023554 17823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:14.026778 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "2f79486791a84d8fb9fa44db4883f19a"
format_stamp: "Formatted at 2026-08-09 22:42:14 on dist-test-slave-rgb1"
server_key: "31420c76621750d24077e88c685e6410"
server_key_iv: "9f006ab862f9c259bddb87ca8f5c2eed"
server_key_version: "encryptionkey@0"
I20260809 22:42:14.027585 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "2f79486791a84d8fb9fa44db4883f19a"
format_stamp: "Formatted at 2026-08-09 22:42:14 on dist-test-slave-rgb1"
server_key: "31420c76621750d24077e88c685e6410"
server_key_iv: "9f006ab862f9c259bddb87ca8f5c2eed"
server_key_version: "encryptionkey@0"
I20260809 22:42:14.033247 17823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 22:42:14.037129 17839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:14.038589 17823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:42:14.038808 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "2f79486791a84d8fb9fa44db4883f19a"
format_stamp: "Formatted at 2026-08-09 22:42:14 on dist-test-slave-rgb1"
server_key: "31420c76621750d24077e88c685e6410"
server_key_iv: "9f006ab862f9c259bddb87ca8f5c2eed"
server_key_version: "encryptionkey@0"
I20260809 22:42:14.039037 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:14.056960 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:14.061823 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:14.062145 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:14.082536 17823 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:34055
I20260809 22:42:14.082607 17894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:34055 every 8 connection(s)
I20260809 22:42:14.083997 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:14.084649 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 17823
I20260809 22:42:14.084875 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:14.085569 29645 external_mini_cluster.cc:1428] Setting key 1b68265c483d7af86a5dc2a642744e3a
I20260809 22:42:14.089344 17895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 22:42:14 dist-test-slave-rgb1 krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:14.104698 17895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a: Bootstrap starting.
I20260809 22:42:14.108820 17895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:14.110152 17895 log.cc:824] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:14.113623 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:14.087860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.28.243.126:34055)
0809 22:42:14.088761 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:14.088773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:14.088820 (+ 47us) server_negotiation.cc:407] Connection header received
0809 22:42:14.090697 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:14.090721 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:14.091119 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:14.091474 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:14.093603 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:14.095095 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:14.097268 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:14.098069 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:14.098225 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:14.104180 (+ 5955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:14.104194 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:14.104211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:14.104263 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:14.108253 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:14.109417 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:14.109425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:14.109433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:14.109636 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:14.110109 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:14.110113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:14.110115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:14.111065 (+ 950us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:14.111448 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:14.111932 (+ 484us) server_negotiation.cc:299] Negotiation successful
0809 22:42:14.112438 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":420,"threads_started":1}
I20260809 22:42:14.113909 17895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a: No bootstrap required, opened a new log
I20260809 22:42:14.122480 17895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79486791a84d8fb9fa44db4883f19a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34055 } }
I20260809 22:42:14.122962 17895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:14.123096 17895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f79486791a84d8fb9fa44db4883f19a, State: Initialized, Role: FOLLOWER
I20260809 22:42:14.123754 17895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79486791a84d8fb9fa44db4883f19a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34055 } }
I20260809 22:42:14.124050 17895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:14.124207 17895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:14.124397 17895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:14.126788 17895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79486791a84d8fb9fa44db4883f19a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34055 } }
I20260809 22:42:14.127334 17895 leader_election.cc:302] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f79486791a84d8fb9fa44db4883f19a; no voters:
I20260809 22:42:14.128100 17895 leader_election.cc:288] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:14.128259 17900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:14.129151 17900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [term 1 LEADER]: Becoming Leader. State: Replica: 2f79486791a84d8fb9fa44db4883f19a, State: Running, Role: LEADER
I20260809 22:42:14.129896 17900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79486791a84d8fb9fa44db4883f19a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34055 } }
I20260809 22:42:14.130451 17895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:14.133674 17901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f79486791a84d8fb9fa44db4883f19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79486791a84d8fb9fa44db4883f19a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34055 } } }
I20260809 22:42:14.134233 17901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:14.134930 17902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f79486791a84d8fb9fa44db4883f19a. Latest consensus state: current_term: 1 leader_uuid: "2f79486791a84d8fb9fa44db4883f19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f79486791a84d8fb9fa44db4883f19a" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34055 } } }
I20260809 22:42:14.135254 17902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:14.141799 17909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:14.147409 17909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:14.157718 17909 catalog_manager.cc:1347] Generated new cluster ID: b8465453117144c6b5b704f6205d296b
I20260809 22:42:14.157873 17909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:14.167484 17909 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:14.168952 17909 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:14.181525 17909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a: Generated new TSK 0
I20260809 22:42:14.182544 17909 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.243.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
Entry for principal kudu/127.28.243.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab.
I20260809 22:42:14.278249 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34055
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:14.641088 17922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:14.641410 17922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:14.641610 17922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:14.649247 17922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:14.649379 17922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:14.649578 17922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:42:14.659704 17922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.65.keytab
--principal=kudu/127.28.243.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:14.661181 17922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:14.662601 17922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:14.676944 17927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:14.676903 17930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:14.676903 17928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:14.677937 17922 server_base.cc:1034] running on GCE node
I20260809 22:42:14.678583 17922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:14.679641 17922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:14.680928 17922 hybrid_clock.cc:648] HybridClock initialized: now 1786315334680824 us; error 61 us; skew 500 ppm
I20260809 22:42:14.681318 17922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:14 dist-test-slave-rgb1 krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315334, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:14.684803 17922 init.cc:377] Logged in from keytab as kudu/127.28.243.65@KRBTEST.COM (short username kudu)
I20260809 22:42:14.686311 17922 webserver.cc:460] Webserver started at http://127.28.243.65:34991/ using document root <none> and password file <none>
I20260809 22:42:14.686973 17922 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:14.687078 17922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:14.687345 17922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:14.690323 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "6c33cfde53144a948417b6238229fa11"
format_stamp: "Formatted at 2026-08-09 22:42:14 on dist-test-slave-rgb1"
server_key: "f6f3ffb1a353f7539f106d319962ef4d"
server_key_iv: "13ce23d725a9373b1e0b5d6d6885e19d"
server_key_version: "encryptionkey@0"
I20260809 22:42:14.691174 17922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "6c33cfde53144a948417b6238229fa11"
format_stamp: "Formatted at 2026-08-09 22:42:14 on dist-test-slave-rgb1"
server_key: "f6f3ffb1a353f7539f106d319962ef4d"
server_key_iv: "13ce23d725a9373b1e0b5d6d6885e19d"
server_key_version: "encryptionkey@0"
I20260809 22:42:14.696846 17922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:42:14.700765 17939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:14.702455 17922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:42:14.702688 17922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "6c33cfde53144a948417b6238229fa11"
format_stamp: "Formatted at 2026-08-09 22:42:14 on dist-test-slave-rgb1"
server_key: "f6f3ffb1a353f7539f106d319962ef4d"
server_key_iv: "13ce23d725a9373b1e0b5d6d6885e19d"
server_key_version: "encryptionkey@0"
I20260809 22:42:14.702976 17922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:14.721689 17922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:14.727046 17922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:14.727392 17922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:14.728657 17922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:14.730233 17922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:14.730366 17922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:14.730535 17922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:14.730618 17922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:14.769706 17922 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:34447
I20260809 22:42:14.769765 18056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:34447 every 8 connection(s)
I20260809 22:42:14.771191 17922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:42:14.773308 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 17922
I20260809 22:42:14.773516 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:42:14.774168 29645 external_mini_cluster.cc:1428] Setting key dcd9d59b8979dd79b53a471bb348c567
Aug 09 22:42:14 dist-test-slave-rgb1 krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315334, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.65@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:14.799363 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:14.774975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:58291 (local address 127.28.243.126:34055)
0809 22:42:14.775572 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:14.775577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:14.776825 (+ 1248us) server_negotiation.cc:407] Connection header received
0809 22:42:14.779719 (+ 2894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:14.779724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:14.779797 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:14.779900 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:14.782163 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:14.782950 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:14.785722 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:14.786160 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:14.786285 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:14.791054 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:14.791060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:14.791064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:14.791110 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:14.795316 (+ 4206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:14.796319 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:14.796329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:14.796336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:14.796535 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:14.797059 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:14.797068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:14.797075 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:14.797572 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:14.797752 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:14.798515 (+ 763us) server_negotiation.cc:299] Negotiation successful
0809 22:42:14.798812 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":314,"threads_started":1}
I20260809 22:42:14.799984 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:14.775520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34055 (local address 127.28.243.65:58291)
0809 22:42:14.776526 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:14.776635 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:14.779367 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:14.780198 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:14.780212 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:14.781071 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:14.781889 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:14.781917 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:14.783164 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:14.783173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:14.785391 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:14.785410 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:14.786435 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:14.786444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:14.786827 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:14.787645 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:42:14.787681 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:14.790738 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:14.795538 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:14.795547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:14.795564 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:14.796140 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:14.796704 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:14.796708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:14.796710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:14.796920 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:14.797711 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:14.797717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:14.798081 (+ 364us) client_negotiation.cc:769] Sending connection context
0809 22:42:14.798318 (+ 237us) client_negotiation.cc:240] Negotiation successful
0809 22:42:14.798761 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":375,"threads_started":1}
I20260809 22:42:14.801811 18057 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34055
I20260809 22:42:14.802172 18057 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:14.803243 18057 heartbeater.cc:511] Master 127.28.243.126:34055 requested a full tablet report, sending...
I20260809 22:42:14.805872 17856 ts_manager.cc:194] Registered new tserver with Master: 6c33cfde53144a948417b6238229fa11 (127.28.243.65:34447)
I20260809 22:42:14.807637 17856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.65@KRBTEST.COM'} at 127.28.243.65:58291
WARNING: no policy specified for kudu/127.28.243.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
Entry for principal kudu/127.28.243.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab.
I20260809 22:42:14.857722 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34055
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:15.226629 18064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:15.226977 18064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:15.227151 18064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:15.235247 18064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:15.235414 18064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:15.235601 18064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:42:15.247278 18064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.66.keytab
--principal=kudu/127.28.243.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:15.248767 18064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:15.250116 18064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:15.264875 18070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:15.264835 18071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:15.264797 18073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:15.265944 18064 server_base.cc:1034] running on GCE node
I20260809 22:42:15.266655 18064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:15.267726 18064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:15.268967 18064 hybrid_clock.cc:648] HybridClock initialized: now 1786315335268912 us; error 52 us; skew 500 ppm
I20260809 22:42:15.269296 18064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:15 dist-test-slave-rgb1 krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315335, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:15.273270 18064 init.cc:377] Logged in from keytab as kudu/127.28.243.66@KRBTEST.COM (short username kudu)
I20260809 22:42:15.275192 18064 webserver.cc:460] Webserver started at http://127.28.243.66:41877/ using document root <none> and password file <none>
I20260809 22:42:15.275970 18064 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:15.276147 18064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:15.276490 18064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:15.279662 18064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "925ab05f103b40169cf7f56066ba49d0"
format_stamp: "Formatted at 2026-08-09 22:42:15 on dist-test-slave-rgb1"
server_key: "7eda2966e77d369a3700d8fd1441115d"
server_key_iv: "6819ec7d7abb263cd1db8a7c29a901c9"
server_key_version: "encryptionkey@0"
I20260809 22:42:15.280495 18064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "925ab05f103b40169cf7f56066ba49d0"
format_stamp: "Formatted at 2026-08-09 22:42:15 on dist-test-slave-rgb1"
server_key: "7eda2966e77d369a3700d8fd1441115d"
server_key_iv: "6819ec7d7abb263cd1db8a7c29a901c9"
server_key_version: "encryptionkey@0"
I20260809 22:42:15.286058 18064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 22:42:15.290002 18080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:15.291549 18064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:42:15.291765 18064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "925ab05f103b40169cf7f56066ba49d0"
format_stamp: "Formatted at 2026-08-09 22:42:15 on dist-test-slave-rgb1"
server_key: "7eda2966e77d369a3700d8fd1441115d"
server_key_iv: "6819ec7d7abb263cd1db8a7c29a901c9"
server_key_version: "encryptionkey@0"
I20260809 22:42:15.292014 18064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:15.323975 18064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:15.329276 18064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:15.329635 18064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:15.330862 18064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:15.332538 18064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:15.332674 18064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 22:42:15.332827 18064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:15.332885 18064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:15.371258 18064 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:33669
I20260809 22:42:15.371357 18200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:33669 every 8 connection(s)
I20260809 22:42:15.372781 18064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:42:15.374001 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 18064
I20260809 22:42:15.374203 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:42:15.374796 29645 external_mini_cluster.cc:1428] Setting key 54f0034ccd571cb01d2af2d73e6b3b77
Aug 09 22:42:15 dist-test-slave-rgb1 krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315335, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.66@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:15.397131 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:15.376787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:53291 (local address 127.28.243.126:34055)
0809 22:42:15.377344 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:15.377352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:15.377954 (+ 602us) server_negotiation.cc:407] Connection header received
0809 22:42:15.379956 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:15.379962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:15.380061 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:15.380193 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:15.381898 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.382668 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.384389 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.384927 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.385081 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:15.389068 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:15.389074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:15.389078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:15.389116 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:15.392083 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:15.393194 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:15.393198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:15.393201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:15.393384 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:15.393973 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:15.393976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:15.393978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:15.394270 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:15.394397 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:15.395636 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0809 22:42:15.396247 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":288,"threads_started":1}
I20260809 22:42:15.397562 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:15.376591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34055 (local address 127.28.243.66:53291)
0809 22:42:15.377789 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:15.377842 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:15.379657 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:15.380463 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:15.380472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:15.381094 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:15.381702 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:15.381720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.382835 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.382843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:15.384203 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:15.384217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.385180 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.385188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:15.385454 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:15.386265 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:42:15.386290 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:15.388795 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:15.392270 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:15.392283 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:15.392305 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:15.393026 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:15.393512 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:15.393520 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:15.393526 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:15.393782 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:15.394403 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:15.394414 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:15.394900 (+ 486us) client_negotiation.cc:769] Sending connection context
0809 22:42:15.395265 (+ 365us) client_negotiation.cc:240] Negotiation successful
0809 22:42:15.395891 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":695,"threads_started":1}
I20260809 22:42:15.399765 18202 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34055
I20260809 22:42:15.400166 18202 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:15.401196 18202 heartbeater.cc:511] Master 127.28.243.126:34055 requested a full tablet report, sending...
I20260809 22:42:15.403293 17856 ts_manager.cc:194] Registered new tserver with Master: 925ab05f103b40169cf7f56066ba49d0 (127.28.243.66:33669)
I20260809 22:42:15.404649 17856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.66@KRBTEST.COM'} at 127.28.243.66:53291
WARNING: no policy specified for kudu/127.28.243.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
Entry for principal kudu/127.28.243.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab.
I20260809 22:42:15.457724 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34055
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:42:15.812173 18057 heartbeater.cc:503] Master 127.28.243.126:34055 was elected leader, sending a full tablet report...
W20260809 22:42:15.819105 18208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:15.819394 18208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:15.819491 18208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:15.826963 18208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:15.827090 18208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:15.827209 18208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:42:15.838163 18208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.67.keytab
--principal=kudu/127.28.243.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:34055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:15.839532 18208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:15.840854 18208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:15.855361 18213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:15.855580 18214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:15.855652 18216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:15.856518 18208 server_base.cc:1034] running on GCE node
I20260809 22:42:15.857355 18208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:15.858649 18208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:15.859927 18208 hybrid_clock.cc:648] HybridClock initialized: now 1786315335859874 us; error 31 us; skew 500 ppm
I20260809 22:42:15.860402 18208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:15 dist-test-slave-rgb1 krb5kdc[17810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315335, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:15.864322 18208 init.cc:377] Logged in from keytab as kudu/127.28.243.67@KRBTEST.COM (short username kudu)
I20260809 22:42:15.866194 18208 webserver.cc:460] Webserver started at http://127.28.243.67:39985/ using document root <none> and password file <none>
I20260809 22:42:15.866955 18208 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:15.867087 18208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:15.867399 18208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:15.871111 18208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "2b5d97a51257409f9db13e399d3d618c"
format_stamp: "Formatted at 2026-08-09 22:42:15 on dist-test-slave-rgb1"
server_key: "0b7531159b5a961e8e74511a904db136"
server_key_iv: "ff50f0bb05b1c30c6ff4b786234371db"
server_key_version: "encryptionkey@0"
I20260809 22:42:15.872082 18208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "2b5d97a51257409f9db13e399d3d618c"
format_stamp: "Formatted at 2026-08-09 22:42:15 on dist-test-slave-rgb1"
server_key: "0b7531159b5a961e8e74511a904db136"
server_key_iv: "ff50f0bb05b1c30c6ff4b786234371db"
server_key_version: "encryptionkey@0"
I20260809 22:42:15.878176 18208 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260809 22:42:15.882228 18223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:15.883836 18208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:42:15.883996 18208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "2b5d97a51257409f9db13e399d3d618c"
format_stamp: "Formatted at 2026-08-09 22:42:15 on dist-test-slave-rgb1"
server_key: "0b7531159b5a961e8e74511a904db136"
server_key_iv: "ff50f0bb05b1c30c6ff4b786234371db"
server_key_version: "encryptionkey@0"
I20260809 22:42:15.884281 18208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:15.914136 18208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:15.919298 18208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:15.919673 18208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:15.920888 18208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:15.922441 18208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:15.922528 18208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:15.922714 18208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:15.922791 18208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:15.962649 18208 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:44385
I20260809 22:42:15.962795 18346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:44385 every 8 connection(s)
I20260809 22:42:15.964187 18208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:42:15.965011 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 18208
I20260809 22:42:15.965235 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:42:15.965857 29645 external_mini_cluster.cc:1428] Setting key 215f1b3fb170bc34a45e7b30ba679b1c
Aug 09 22:42:15 dist-test-slave-rgb1 krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315335, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.67@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:15.986544 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:15.967248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:37195 (local address 127.28.243.126:34055)
0809 22:42:15.967802 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:15.967809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:15.969048 (+ 1239us) server_negotiation.cc:407] Connection header received
0809 22:42:15.971095 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:15.971100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:15.971166 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:15.971284 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:15.972936 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.973736 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.975328 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.975735 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.975836 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:15.979702 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:15.979707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:15.979710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:15.979766 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:15.983283 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:15.984096 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:15.984100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:15.984103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:15.984231 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:15.984767 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:15.984771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:15.984773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:15.985171 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:15.985319 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:15.986069 (+ 750us) server_negotiation.cc:299] Negotiation successful
0809 22:42:15.986268 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":260,"threads_started":1}
I20260809 22:42:15.987619 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:15.967957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:34055 (local address 127.28.243.67:37195)
0809 22:42:15.968864 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:15.968924 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:15.970786 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:15.971518 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:15.971527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:15.972113 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:15.972747 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:15.972766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.973894 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.973898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:15.975173 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:15.975185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:15.975930 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:15.975934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:15.976187 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:15.976932 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:42:15.976954 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:15.979434 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:15.983434 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:15.983442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:15.983456 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:15.983955 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:15.984353 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:42:15.984357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:42:15.984360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:42:15.984630 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:42:15.985304 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:15.985311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:15.985656 (+ 345us) client_negotiation.cc:769] Sending connection context
0809 22:42:15.985892 (+ 236us) client_negotiation.cc:240] Negotiation successful
0809 22:42:15.986326 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":373,"threads_started":1}
I20260809 22:42:15.989323 18347 heartbeater.cc:347] Connected to a master server at 127.28.243.126:34055
I20260809 22:42:15.989642 18347 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:15.990368 18347 heartbeater.cc:511] Master 127.28.243.126:34055 requested a full tablet report, sending...
I20260809 22:42:15.991832 17856 ts_manager.cc:194] Registered new tserver with Master: 2b5d97a51257409f9db13e399d3d618c (127.28.243.67:44385)
I20260809 22:42:15.992743 17856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.243.67@KRBTEST.COM'} at 127.28.243.67:37195
I20260809 22:42:15.995153 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:42:16.014686 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:15.999700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.28.243.126:34055)
0809 22:42:15.999917 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:15.999924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.000631 (+ 707us) server_negotiation.cc:407] Connection header received
0809 22:42:16.000762 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.000766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.000836 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.000925 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:16.002711 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.003541 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.005261 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.005705 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.005832 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.008542 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:16.008547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:16.008610 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:16.008665 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:16.011598 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.012359 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.012363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.012366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.012508 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.013115 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.013119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.013121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.013516 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:16.013735 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.014175 (+ 440us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.014423 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 22:42:16.019158 17856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:42:16.026296 17856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:42:16.052165 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.039857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.28.243.65:34447)
0809 22:42:16.042046 (+ 2189us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.042060 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.042096 (+ 36us) server_negotiation.cc:407] Connection header received
0809 22:42:16.042219 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.042230 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.042538 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.043695 (+ 1157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:16.044039 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.046219 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.049222 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.050834 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.050983 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.051246 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.051634 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.051824 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1880,"thread_start_us":1691,"threads_started":1}
I20260809 22:42:16.052275 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.039417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:34447 (local address 127.0.0.1:57718)
0809 22:42:16.040218 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:16.040270 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:16.040458 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:16.042831 (+ 2373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:16.042843 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:16.042905 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:16.043384 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.043401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.046468 (+ 3067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.046477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.049015 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.049030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.051059 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.051066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.051326 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.051412 (+ 86us) client_negotiation.cc:769] Sending connection context
0809 22:42:16.051580 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 22:42:16.051794 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":3,"thread_start_us":323,"threads_started":1}
I20260809 22:42:16.055548 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.040214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:33669 (local address 127.0.0.1:52364)
0809 22:42:16.041120 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:16.041141 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:16.041272 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:16.047557 (+ 6285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:16.047562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:16.047592 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:16.048100 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.048113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.050068 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.050074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.052222 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.052240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.054519 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.054527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.054788 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.054828 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 22:42:16.054914 (+ 86us) client_negotiation.cc:240] Negotiation successful
0809 22:42:16.055069 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":53,"thread_start_us":807,"threads_started":1}
I20260809 22:42:16.055655 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.040845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.28.243.66:33669)
0809 22:42:16.046904 (+ 6059us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.046914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.046949 (+ 35us) server_negotiation.cc:407] Connection header received
0809 22:42:16.047059 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.047069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.047331 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.047540 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:16.048311 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.049892 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.052469 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.054238 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.054798 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.055077 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.055160 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.055335 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5868,"thread_start_us":5751,"threads_started":1}
I20260809 22:42:16.056015 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.041189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.28.243.67:44385)
0809 22:42:16.047312 (+ 6123us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.047319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.047352 (+ 33us) server_negotiation.cc:407] Connection header received
0809 22:42:16.047466 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.047475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.047685 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.047881 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:16.048719 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.050273 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.052845 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.054160 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.054303 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.054543 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.055454 (+ 911us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.055643 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5961,"thread_start_us":5790,"threads_started":1}
I20260809 22:42:16.056355 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.041674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:44385 (local address 127.0.0.1:46170)
0809 22:42:16.042207 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:16.042229 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:16.042342 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:16.047883 (+ 5541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:16.047890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:16.047919 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:16.048413 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.048428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.050431 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.050436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.052669 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.052685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.054708 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.054721 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.054954 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.054981 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 22:42:16.055050 (+ 69us) client_negotiation.cc:240] Negotiation successful
0809 22:42:16.055155 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":5,"thread_start_us":275,"threads_started":1}
I20260809 22:42:16.057277 17991 tablet_service.cc:1467] Processing CreateTablet for tablet eaefe0e25527470297a648202c223f1d (DEFAULT_TABLE table=test-table [id=e389a6dfce88401a9e906422bd081aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:16.058967 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaefe0e25527470297a648202c223f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:16.059681 18276 tablet_service.cc:1467] Processing CreateTablet for tablet eaefe0e25527470297a648202c223f1d (DEFAULT_TABLE table=test-table [id=e389a6dfce88401a9e906422bd081aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:16.059713 18135 tablet_service.cc:1467] Processing CreateTablet for tablet eaefe0e25527470297a648202c223f1d (DEFAULT_TABLE table=test-table [id=e389a6dfce88401a9e906422bd081aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:16.061096 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaefe0e25527470297a648202c223f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:16.061342 18276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaefe0e25527470297a648202c223f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:16.072598 18366 tablet_bootstrap.cc:492] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Bootstrap starting.
I20260809 22:42:16.076515 18366 tablet_bootstrap.cc:654] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:16.077653 18366 log.cc:824] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:16.080271 18367 tablet_bootstrap.cc:492] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Bootstrap starting.
I20260809 22:42:16.080742 18366 tablet_bootstrap.cc:492] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: No bootstrap required, opened a new log
I20260809 22:42:16.081012 18366 ts_tablet_manager.cc:1397] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 22:42:16.083693 18367 tablet_bootstrap.cc:654] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:16.085136 18367 log.cc:824] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:16.088699 18368 tablet_bootstrap.cc:492] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Bootstrap starting.
I20260809 22:42:16.089397 18367 tablet_bootstrap.cc:492] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: No bootstrap required, opened a new log
I20260809 22:42:16.089745 18367 ts_tablet_manager.cc:1397] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260809 22:42:16.092666 18368 tablet_bootstrap.cc:654] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:16.094199 18368 log.cc:824] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:16.095278 18366 raft_consensus.cc:348] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.095870 18366 raft_consensus.cc:374] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:16.096029 18366 raft_consensus.cc:729] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5d97a51257409f9db13e399d3d618c, State: Initialized, Role: FOLLOWER
I20260809 22:42:16.096938 18366 consensus_queue.cc:260] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.104809 18368 tablet_bootstrap.cc:492] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: No bootstrap required, opened a new log
I20260809 22:42:16.105150 18368 ts_tablet_manager.cc:1397] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.009s
I20260809 22:42:16.105609 18366 ts_tablet_manager.cc:1428] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Time spent starting tablet: real 0.024s user 0.015s sys 0.004s
I20260809 22:42:16.105772 18367 raft_consensus.cc:348] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.106112 18347 heartbeater.cc:503] Master 127.28.243.126:34055 was elected leader, sending a full tablet report...
I20260809 22:42:16.106277 18367 raft_consensus.cc:374] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:16.106391 18367 raft_consensus.cc:729] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c33cfde53144a948417b6238229fa11, State: Initialized, Role: FOLLOWER
I20260809 22:42:16.107221 18367 consensus_queue.cc:260] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.115216 18367 ts_tablet_manager.cc:1428] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Time spent starting tablet: real 0.025s user 0.016s sys 0.001s
I20260809 22:42:16.117494 18368 raft_consensus.cc:348] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.117971 18368 raft_consensus.cc:374] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:16.118096 18368 raft_consensus.cc:729] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925ab05f103b40169cf7f56066ba49d0, State: Initialized, Role: FOLLOWER
I20260809 22:42:16.119050 18368 consensus_queue.cc:260] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.120383 18202 heartbeater.cc:503] Master 127.28.243.126:34055 was elected leader, sending a full tablet report...
I20260809 22:42:16.121071 18368 ts_tablet_manager.cc:1428] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260809 22:42:16.125243 18203 tablet.cc:2369] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:16.146234 18372 raft_consensus.cc:482] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:42:16.146596 18372 raft_consensus.cc:504] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.148800 18372 leader_election.cc:288] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c33cfde53144a948417b6238229fa11 (127.28.243.65:34447), 925ab05f103b40169cf7f56066ba49d0 (127.28.243.66:33669)
I20260809 22:42:16.155537 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.149449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:33669 (local address 127.28.243.67:60509)
0809 22:42:16.149664 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:16.149688 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:16.149874 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:16.150441 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:16.150453 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:16.150493 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:16.150777 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.150790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.151879 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.151889 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.153490 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.153509 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.154752 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.154758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.155051 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.155098 (+ 47us) client_negotiation.cc:769] Sending connection context
0809 22:42:16.155191 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 22:42:16.155304 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":21}
I20260809 22:42:16.155797 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.149445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:60509 (local address 127.28.243.66:33669)
0809 22:42:16.149677 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.149681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.149799 (+ 118us) server_negotiation.cc:407] Connection header received
0809 22:42:16.150053 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.150059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.150156 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.150270 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:16.150958 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.151710 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.153679 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.154586 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.154716 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.154774 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.155291 (+ 517us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.155519 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260809 22:42:16.157018 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefe0e25527470297a648202c223f1d" candidate_uuid: "2b5d97a51257409f9db13e399d3d618c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ab05f103b40169cf7f56066ba49d0" is_pre_election: true
I20260809 22:42:16.157548 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.149445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:51791 (local address 127.28.243.65:34447)
0809 22:42:16.149821 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.149826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.151135 (+ 1309us) server_negotiation.cc:407] Connection header received
0809 22:42:16.151335 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.151340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.151403 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.151509 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:16.152258 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.153184 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.155236 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.156547 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.156744 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.156795 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.157171 (+ 376us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.157277 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260809 22:42:16.157681 18375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.149447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:34447 (local address 127.28.243.67:51791)
0809 22:42:16.151007 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:16.151024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:16.151175 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:16.151524 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:16.151530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:16.151551 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:16.152045 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.152056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.153347 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.153352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.154978 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:16.154993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.156797 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.156803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:16.156980 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.157019 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 22:42:16.157124 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 22:42:16.157274 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"mutex_wait_us":610,"thread_start_us":512,"threads_started":1}
I20260809 22:42:16.157713 18155 raft_consensus.cc:2454] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5d97a51257409f9db13e399d3d618c in term 0.
I20260809 22:42:16.158627 18226 leader_election.cc:302] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5d97a51257409f9db13e399d3d618c, 925ab05f103b40169cf7f56066ba49d0; no voters:
I20260809 22:42:16.158874 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefe0e25527470297a648202c223f1d" candidate_uuid: "2b5d97a51257409f9db13e399d3d618c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33cfde53144a948417b6238229fa11" is_pre_election: true
I20260809 22:42:16.159126 18372 raft_consensus.cc:2781] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:42:16.159385 18372 raft_consensus.cc:482] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:42:16.159521 18372 raft_consensus.cc:3037] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:16.159469 18012 raft_consensus.cc:2454] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b5d97a51257409f9db13e399d3d618c in term 0.
I20260809 22:42:16.162531 18372 raft_consensus.cc:504] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.163272 18372 leader_election.cc:288] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [CANDIDATE]: Term 1 election: Requested vote from peers 6c33cfde53144a948417b6238229fa11 (127.28.243.65:34447), 925ab05f103b40169cf7f56066ba49d0 (127.28.243.66:33669)
I20260809 22:42:16.163975 18012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefe0e25527470297a648202c223f1d" candidate_uuid: "2b5d97a51257409f9db13e399d3d618c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33cfde53144a948417b6238229fa11"
I20260809 22:42:16.164296 18012 raft_consensus.cc:3037] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:16.164266 18155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaefe0e25527470297a648202c223f1d" candidate_uuid: "2b5d97a51257409f9db13e399d3d618c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "925ab05f103b40169cf7f56066ba49d0"
I20260809 22:42:16.164510 18155 raft_consensus.cc:3037] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:16.166843 18155 raft_consensus.cc:2454] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b5d97a51257409f9db13e399d3d618c in term 1.
I20260809 22:42:16.167027 18012 raft_consensus.cc:2454] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b5d97a51257409f9db13e399d3d618c in term 1.
I20260809 22:42:16.167448 18226 leader_election.cc:302] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b5d97a51257409f9db13e399d3d618c, 925ab05f103b40169cf7f56066ba49d0; no voters:
I20260809 22:42:16.167832 18372 raft_consensus.cc:2781] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:16.168403 18372 raft_consensus.cc:686] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 1 LEADER]: Becoming Leader. State: Replica: 2b5d97a51257409f9db13e399d3d618c, State: Running, Role: LEADER
I20260809 22:42:16.169003 18372 consensus_queue.cc:237] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } }
I20260809 22:42:16.174614 17856 catalog_manager.cc:5495] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b5d97a51257409f9db13e399d3d618c (127.28.243.67). New cstate: current_term: 1 leader_uuid: "2b5d97a51257409f9db13e399d3d618c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b5d97a51257409f9db13e399d3d618c" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 44385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 } health_report { overall_health: UNKNOWN } } }
I20260809 22:42:16.214612 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.208745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.28.243.67:44385)
0809 22:42:16.209021 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.209028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.209048 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:42:16.209309 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.209314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.209396 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.209617 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:42:16.210018 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.210778 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.212195 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.212699 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.212847 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.213418 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:42:16.213960 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:42:16.214130 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.214237 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.214327 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260809 22:42:16.216120 18348 tablet.cc:2369] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:16.223210 18155 raft_consensus.cc:1261] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 1 FOLLOWER]: Refusing update from remote peer 2b5d97a51257409f9db13e399d3d618c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:16.223205 18012 raft_consensus.cc:1261] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 1 FOLLOWER]: Refusing update from remote peer 2b5d97a51257409f9db13e399d3d618c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:16.224192 18372 consensus_queue.cc:1035] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [LEADER]: Connected to new peer: Peer: permanent_uuid: "925ab05f103b40169cf7f56066ba49d0" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 33669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:16.224515 18376 consensus_queue.cc:1035] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c33cfde53144a948417b6238229fa11" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:16.252408 17855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:16.252851 17855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:16.257165 17855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f79486791a84d8fb9fa44db4883f19a: Sending DeleteTablet for 3 replicas of tablet eaefe0e25527470297a648202c223f1d
I20260809 22:42:16.258546 17991 tablet_service.cc:1514] Processing DeleteTablet for tablet eaefe0e25527470297a648202c223f1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:16 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:57718
I20260809 22:42:16.258872 18276 tablet_service.cc:1514] Processing DeleteTablet for tablet eaefe0e25527470297a648202c223f1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:16 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:46170
I20260809 22:42:16.259686 18135 tablet_service.cc:1514] Processing DeleteTablet for tablet eaefe0e25527470297a648202c223f1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:16 UTC) from {username='kudu', principal='kudu/127.28.243.126@KRBTEST.COM'} at 127.0.0.1:52364
I20260809 22:42:16.260223 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 17922
I20260809 22:42:16.273242 18393 tablet_replica.cc:330] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: stopping tablet replica
I20260809 22:42:16.276695 18392 tablet_replica.cc:330] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: stopping tablet replica
I20260809 22:42:16.277212 18393 raft_consensus.cc:2229] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:16.277305 18392 raft_consensus.cc:2229] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:42:16.277940 18393 raft_consensus.cc:2258] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:16.278539 18392 raft_consensus.cc:2258] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:16.281921 18392 ts_tablet_manager.cc:1905] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:16.283946 18393 ts_tablet_manager.cc:1905] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:16.285605 18391 tablet_replica.cc:330] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: stopping tablet replica
I20260809 22:42:16.286053 18391 raft_consensus.cc:2229] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:16.286635 18391 raft_consensus.cc:2258] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:16.288790 18391 ts_tablet_manager.cc:1905] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:16.293851 18392 ts_tablet_manager.cc:1918] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:16.294167 18392 log.cc:1196] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/eaefe0e25527470297a648202c223f1d
I20260809 22:42:16.294921 18392 ts_tablet_manager.cc:1939] T eaefe0e25527470297a648202c223f1d P 2b5d97a51257409f9db13e399d3d618c: Deleting consensus metadata
Aug 09 22:42:16 dist-test-slave-rgb1 krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.65@KRBTEST.COM
I20260809 22:42:16.296965 18391 ts_tablet_manager.cc:1918] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:16.297212 18391 log.cc:1196] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/eaefe0e25527470297a648202c223f1d
I20260809 22:42:16.297932 18391 ts_tablet_manager.cc:1939] T eaefe0e25527470297a648202c223f1d P 6c33cfde53144a948417b6238229fa11: Deleting consensus metadata
I20260809 22:42:16.297817 17841 catalog_manager.cc:4841] TS 2b5d97a51257409f9db13e399d3d618c (127.28.243.67:44385): tablet eaefe0e25527470297a648202c223f1d (table test-table [id=e389a6dfce88401a9e906422bd081aa4]) successfully deleted
I20260809 22:42:16.300496 17841 catalog_manager.cc:4841] TS 6c33cfde53144a948417b6238229fa11 (127.28.243.65:34447): tablet eaefe0e25527470297a648202c223f1d (table test-table [id=e389a6dfce88401a9e906422bd081aa4]) successfully deleted
I20260809 22:42:16.301208 18393 ts_tablet_manager.cc:1918] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:16.301403 18393 log.cc:1196] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/eaefe0e25527470297a648202c223f1d
I20260809 22:42:16.301987 18393 ts_tablet_manager.cc:1939] T eaefe0e25527470297a648202c223f1d P 925ab05f103b40169cf7f56066ba49d0: Deleting consensus metadata
I20260809 22:42:16.303889 17842 catalog_manager.cc:4841] TS 925ab05f103b40169cf7f56066ba49d0 (127.28.243.66:33669): tablet eaefe0e25527470297a648202c223f1d (table test-table [id=e389a6dfce88401a9e906422bd081aa4]) successfully deleted
I20260809 22:42:16.305537 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.261761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.28.243.65:34447)
0809 22:42:16.261981 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.261989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.272217 (+ 10228us) server_negotiation.cc:407] Connection header received
0809 22:42:16.273907 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.273915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.274121 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.274218 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:16.278459 (+ 4241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.279737 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.284702 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.285507 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.285625 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.297724 (+ 12099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:16.297741 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:16.297760 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:16.297820 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:16.302034 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.302940 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.302950 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.302954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.303110 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.303513 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.303518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.303521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.303999 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:16.304289 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.304886 (+ 597us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.305141 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 22:42:16.306209 18052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:16.458912 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 17922
I20260809 22:42:16.480584 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 18064
Aug 09 22:42:16 dist-test-slave-rgb1 krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.66@KRBTEST.COM
I20260809 22:42:16.501375 18363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.481682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.28.243.66:33669)
0809 22:42:16.481904 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.481910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.481933 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:42:16.482004 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.482011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.482095 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.482212 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:16.483704 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.485053 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.486458 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.487105 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.487252 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.493173 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:16.493194 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:16.493215 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:16.493275 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:16.497676 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.498699 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.498713 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.498717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.498889 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.499407 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.499413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.499416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.500012 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:16.500331 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.500730 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.500977 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:42:16.502269 18196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:16.603541 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 18064
I20260809 22:42:16.624984 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 18208
Aug 09 22:42:16 dist-test-slave-rgb1 krb5kdc[17810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.67@KRBTEST.COM
I20260809 22:42:16.644232 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:16.625887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.28.243.67:44385)
0809 22:42:16.626125 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:16.626132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:16.626156 (+ 24us) server_negotiation.cc:407] Connection header received
0809 22:42:16.626232 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:16.626239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:16.626359 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:16.626493 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:16.627930 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.629197 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.630572 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:16.631308 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:16.631511 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:16.636955 (+ 5444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:16.636974 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:16.636991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:16.637043 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:16.640911 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.641750 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.641758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.641760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.641958 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:16.642400 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:16.642407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:16.642410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:16.642924 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:16.643277 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:16.643574 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 22:42:16.643820 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 22:42:16.645040 18342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:16.772492 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 18208
I20260809 22:42:16.796185 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 17823
I20260809 22:42:16.797636 17890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:16.913278 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 17823
2026-08-09T22:42:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4904 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T22:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:16Z Disabled control of system clock
I20260809 22:42:17.015338 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33391
--webserver_interface=127.28.243.126
--webserver_port=0
--builtin_ntp_servers=127.28.243.84:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 22:42:17.380908 18402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:17.381215 18402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:17.381300 18402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:17.388902 18402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:17.389027 18402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:17.389067 18402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:17.389120 18402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:17.389181 18402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:17.402348 18402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:17.403857 18402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:17.405828 18402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:17.418566 18411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:17.418505 18410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:17.419051 18413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:17.419958 18402 server_base.cc:1034] running on GCE node
I20260809 22:42:17.420909 18402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:17.422429 18402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:17.423712 18402 hybrid_clock.cc:648] HybridClock initialized: now 1786315337423654 us; error 55 us; skew 500 ppm
I20260809 22:42:17.424124 18402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:17.426802 18402 webserver.cc:460] Webserver started at http://127.28.243.126:38701/ using document root <none> and password file <none>
I20260809 22:42:17.427559 18402 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:17.427702 18402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:17.428058 18402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:17.431248 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "a9f8cead85fb41e3bbd534035bb913d5"
format_stamp: "Formatted at 2026-08-09 22:42:17 on dist-test-slave-rgb1"
server_key: "f75751c8d476ce9cf9b429d39f26c171"
server_key_iv: "b719e2a12f2afb59e812cb90d8920e02"
server_key_version: "encryptionkey@0"
I20260809 22:42:17.432111 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "a9f8cead85fb41e3bbd534035bb913d5"
format_stamp: "Formatted at 2026-08-09 22:42:17 on dist-test-slave-rgb1"
server_key: "f75751c8d476ce9cf9b429d39f26c171"
server_key_iv: "b719e2a12f2afb59e812cb90d8920e02"
server_key_version: "encryptionkey@0"
I20260809 22:42:17.437862 18402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 22:42:17.441931 18419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:17.443373 18402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:42:17.443602 18402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "a9f8cead85fb41e3bbd534035bb913d5"
format_stamp: "Formatted at 2026-08-09 22:42:17 on dist-test-slave-rgb1"
server_key: "f75751c8d476ce9cf9b429d39f26c171"
server_key_iv: "b719e2a12f2afb59e812cb90d8920e02"
server_key_version: "encryptionkey@0"
I20260809 22:42:17.443841 18402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:17.457209 18402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:17.458194 18402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:17.458518 18402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:17.479830 18402 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33391
I20260809 22:42:17.479859 18471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33391 every 8 connection(s)
I20260809 22:42:17.481312 18402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:17.485481 18472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:17.490227 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 18402
I20260809 22:42:17.490453 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:17.490917 29645 external_mini_cluster.cc:1428] Setting key dd7d7be2fe5ce4b6d39e03f9b50ceb5b
I20260809 22:42:17.498418 18472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5: Bootstrap starting.
I20260809 22:42:17.502933 18472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:17.504490 18472 log.cc:824] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:17.508199 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:17.493092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.28.243.126:33391)
0809 22:42:17.494220 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:17.494235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:17.494290 (+ 55us) server_negotiation.cc:407] Connection header received
0809 22:42:17.496835 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:17.496861 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:17.497336 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:17.497752 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:17.498343 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:17.499972 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:17.502273 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:17.503042 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:17.503273 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:17.504935 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:17.504948 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:17.504965 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:17.505999 (+ 1034us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:17.506137 (+ 138us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:17.506173 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:17.506337 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:17.506656 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 22:42:17.507080 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":444,"threads_started":1}
I20260809 22:42:17.508370 18472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5: No bootstrap required, opened a new log
I20260809 22:42:17.516949 18472 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8cead85fb41e3bbd534035bb913d5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33391 } }
I20260809 22:42:17.517401 18472 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:17.517516 18472 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f8cead85fb41e3bbd534035bb913d5, State: Initialized, Role: FOLLOWER
I20260809 22:42:17.518113 18472 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8cead85fb41e3bbd534035bb913d5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33391 } }
I20260809 22:42:17.518379 18472 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:17.518525 18472 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:17.518711 18472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:17.521270 18472 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8cead85fb41e3bbd534035bb913d5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33391 } }
I20260809 22:42:17.521823 18472 leader_election.cc:302] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9f8cead85fb41e3bbd534035bb913d5; no voters:
I20260809 22:42:17.522555 18472 leader_election.cc:288] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:17.522737 18477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:17.523597 18477 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [term 1 LEADER]: Becoming Leader. State: Replica: a9f8cead85fb41e3bbd534035bb913d5, State: Running, Role: LEADER
I20260809 22:42:17.524189 18477 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8cead85fb41e3bbd534035bb913d5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33391 } }
I20260809 22:42:17.524955 18472 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:17.528174 18479 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f8cead85fb41e3bbd534035bb913d5. Latest consensus state: current_term: 1 leader_uuid: "a9f8cead85fb41e3bbd534035bb913d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8cead85fb41e3bbd534035bb913d5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33391 } } }
I20260809 22:42:17.528363 18478 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9f8cead85fb41e3bbd534035bb913d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f8cead85fb41e3bbd534035bb913d5" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33391 } } }
I20260809 22:42:17.528822 18479 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:17.528918 18478 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:17.530007 18485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:17.537333 18485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:17.547909 18485 catalog_manager.cc:1347] Generated new cluster ID: 4c5935463d9246c3b6de43c3036677c0
I20260809 22:42:17.548178 18485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:17.570103 18485 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:17.571497 18485 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:17.591040 18485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5: Generated new TSK 0
I20260809 22:42:17.591987 18485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:17.599434 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.65:0
--local_ip_for_outbound_sockets=127.28.243.65
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33391
--builtin_ntp_servers=127.28.243.84:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 22:42:17.984836 18497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:17.985239 18497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:17.985412 18497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:17.994199 18497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:17.994381 18497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:17.994585 18497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.65
I20260809 22:42:18.005558 18497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.243.65
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.65
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:18.007287 18497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:18.009517 18497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:18.025615 18506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:18.025554 18504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:18.025996 18503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:18.027197 18497 server_base.cc:1034] running on GCE node
I20260809 22:42:18.027879 18497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:18.029110 18497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:18.030366 18497 hybrid_clock.cc:648] HybridClock initialized: now 1786315338030305 us; error 59 us; skew 500 ppm
I20260809 22:42:18.030733 18497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:18.033416 18497 webserver.cc:460] Webserver started at http://127.28.243.65:35767/ using document root <none> and password file <none>
I20260809 22:42:18.034179 18497 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:18.034322 18497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:18.034710 18497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:18.037869 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data/instance:
uuid: "f7effac46ff34d1a954527a5b17da4cf"
format_stamp: "Formatted at 2026-08-09 22:42:18 on dist-test-slave-rgb1"
server_key: "e0d1902a5bc9326f5147de7fffce555f"
server_key_iv: "a96fb8d4dcd6624c24a83c79286913a8"
server_key_version: "encryptionkey@0"
I20260809 22:42:18.038858 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance:
uuid: "f7effac46ff34d1a954527a5b17da4cf"
format_stamp: "Formatted at 2026-08-09 22:42:18 on dist-test-slave-rgb1"
server_key: "e0d1902a5bc9326f5147de7fffce555f"
server_key_iv: "a96fb8d4dcd6624c24a83c79286913a8"
server_key_version: "encryptionkey@0"
I20260809 22:42:18.044461 18497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:42:18.048382 18513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:18.049965 18497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:18.050215 18497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal
uuid: "f7effac46ff34d1a954527a5b17da4cf"
format_stamp: "Formatted at 2026-08-09 22:42:18 on dist-test-slave-rgb1"
server_key: "e0d1902a5bc9326f5147de7fffce555f"
server_key_iv: "a96fb8d4dcd6624c24a83c79286913a8"
server_key_version: "encryptionkey@0"
I20260809 22:42:18.050470 18497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:18.067126 18497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:18.067958 18497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:18.068248 18497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:18.069444 18497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:18.070974 18497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:18.071066 18497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:18.071146 18497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:18.071203 18497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:18.110041 18497 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.65:33639
I20260809 22:42:18.110103 18639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.65:33639 every 8 connection(s)
I20260809 22:42:18.111614 18497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/data/info.pb
I20260809 22:42:18.120496 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 18497
I20260809 22:42:18.120796 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal/instance
I20260809 22:42:18.121266 29645 external_mini_cluster.cc:1428] Setting key cafbba0071e318457b6df455d5e47f75
I20260809 22:42:18.124302 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.66:0
--local_ip_for_outbound_sockets=127.28.243.66
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33391
--builtin_ntp_servers=127.28.243.84:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 22:42:18.125734 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:18.114598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:55229 (local address 127.28.243.126:33391)
0809 22:42:18.115156 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:18.115160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:18.116135 (+ 975us) server_negotiation.cc:407] Connection header received
0809 22:42:18.118109 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:18.118115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:18.118227 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:18.118363 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:18.119489 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.120224 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.122250 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.122690 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.122849 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:18.124714 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:18.124720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:18.124723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:18.124745 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:18.124789 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:18.124802 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:18.124998 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:18.125217 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 22:42:18.125336 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":289,"threads_started":1}
I20260809 22:42:18.126999 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:18.115011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33391 (local address 127.28.243.65:55229)
0809 22:42:18.115947 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:18.115991 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:18.117783 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:18.118625 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:18.118635 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:18.118728 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:18.119312 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:18.119329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.120403 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.120407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:18.122089 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:18.122113 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.122959 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.122967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:18.123205 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:18.124187 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:18.124190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:18.124234 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:18.124339 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:18.124928 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:18.124942 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:18.124951 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 22:42:18.125191 (+ 240us) client_negotiation.cc:240] Negotiation successful
0809 22:42:18.125726 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":281,"threads_started":1}
I20260809 22:42:18.128846 18640 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33391
I20260809 22:42:18.129215 18640 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:18.130014 18640 heartbeater.cc:511] Master 127.28.243.126:33391 requested a full tablet report, sending...
I20260809 22:42:18.132731 18436 ts_manager.cc:194] Registered new tserver with Master: f7effac46ff34d1a954527a5b17da4cf (127.28.243.65:33639)
I20260809 22:42:18.135290 18436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.65:55229
W20260809 22:42:18.493619 18645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:18.493970 18645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:18.494148 18645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:18.501937 18645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:18.502103 18645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:18.502285 18645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.66
I20260809 22:42:18.512264 18645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.243.66
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.66
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:18.513706 18645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:18.515578 18645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:18.529745 18650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:18.529803 18654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:18.529713 18651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:18.531136 18645 server_base.cc:1034] running on GCE node
I20260809 22:42:18.531884 18645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:18.533033 18645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:18.534319 18645 hybrid_clock.cc:648] HybridClock initialized: now 1786315338534251 us; error 65 us; skew 500 ppm
I20260809 22:42:18.534698 18645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:18.537392 18645 webserver.cc:460] Webserver started at http://127.28.243.66:39317/ using document root <none> and password file <none>
I20260809 22:42:18.538151 18645 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:18.538301 18645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:18.538681 18645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:18.541918 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data/instance:
uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2"
format_stamp: "Formatted at 2026-08-09 22:42:18 on dist-test-slave-rgb1"
server_key: "de217fc652a4dd266682ea02f4d5bffd"
server_key_iv: "2a9a7fc918e6d9ae2391746f916e1eeb"
server_key_version: "encryptionkey@0"
I20260809 22:42:18.542757 18645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance:
uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2"
format_stamp: "Formatted at 2026-08-09 22:42:18 on dist-test-slave-rgb1"
server_key: "de217fc652a4dd266682ea02f4d5bffd"
server_key_iv: "2a9a7fc918e6d9ae2391746f916e1eeb"
server_key_version: "encryptionkey@0"
I20260809 22:42:18.548750 18645 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:42:18.552809 18660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:18.554744 18645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:42:18.554952 18645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal
uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2"
format_stamp: "Formatted at 2026-08-09 22:42:18 on dist-test-slave-rgb1"
server_key: "de217fc652a4dd266682ea02f4d5bffd"
server_key_iv: "2a9a7fc918e6d9ae2391746f916e1eeb"
server_key_version: "encryptionkey@0"
I20260809 22:42:18.555223 18645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:18.577526 18645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:18.578512 18645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:18.578878 18645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:18.580001 18645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:18.581866 18645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:18.582002 18645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:18.582151 18645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:18.582262 18645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:18.621650 18645 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.66:42605
I20260809 22:42:18.621692 18781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.66:42605 every 8 connection(s)
I20260809 22:42:18.623164 18645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/data/info.pb
I20260809 22:42:18.632102 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 18645
I20260809 22:42:18.632313 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal/instance
I20260809 22:42:18.632963 29645 external_mini_cluster.cc:1428] Setting key f40b55ec788ef70c4ca8c028deff95d7
I20260809 22:42:18.636469 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.243.67:0
--local_ip_for_outbound_sockets=127.28.243.67
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33391
--builtin_ntp_servers=127.28.243.84:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 22:42:18.638199 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:18.626090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.66:42075 (local address 127.28.243.126:33391)
0809 22:42:18.626679 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:18.626684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:18.627906 (+ 1222us) server_negotiation.cc:407] Connection header received
0809 22:42:18.629977 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:18.629984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:18.630095 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:18.630227 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:18.631426 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.632714 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.634602 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.635282 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.635393 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:18.637308 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:18.637314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:18.637317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:18.637338 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:18.637381 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:18.637394 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:18.637482 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:18.637793 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 22:42:18.637880 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":469,"thread_start_us":305,"threads_started":1}
I20260809 22:42:18.639716 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:18.626731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33391 (local address 127.28.243.66:42075)
0809 22:42:18.627735 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:18.627774 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:18.629670 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:18.630485 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:18.630495 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:18.630617 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:18.631240 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:18.631259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.632954 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.632959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:18.634416 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:18.634430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:18.635552 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:18.635560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:18.635874 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:18.636887 (+ 1013us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:18.636892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:18.636953 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:18.637072 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:18.637507 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:18.637520 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:18.637532 (+ 12us) client_negotiation.cc:769] Sending connection context
0809 22:42:18.637741 (+ 209us) client_negotiation.cc:240] Negotiation successful
0809 22:42:18.638225 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":340,"threads_started":1}
I20260809 22:42:18.641425 18782 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33391
I20260809 22:42:18.641753 18782 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:18.642604 18782 heartbeater.cc:511] Master 127.28.243.126:33391 requested a full tablet report, sending...
I20260809 22:42:18.644440 18436 ts_manager.cc:194] Registered new tserver with Master: 0b6dac4c3ee4436aa1d0927cdff5baf2 (127.28.243.66:42605)
I20260809 22:42:18.645481 18436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.66:42075
W20260809 22:42:19.000655 18786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:19.000988 18786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:19.001214 18786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:19.009071 18786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:19.009202 18786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:19.009313 18786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.243.67
I20260809 22:42:19.019618 18786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.243.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.243.67
--webserver_port=0
--tserver_master_addrs=127.28.243.126:33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.243.67
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:19.021215 18786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:19.023141 18786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:19.037928 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:19.037906 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:19.038522 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:19.039436 18786 server_base.cc:1034] running on GCE node
I20260809 22:42:19.040136 18786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:19.041419 18786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:19.042636 18786 hybrid_clock.cc:648] HybridClock initialized: now 1786315339042561 us; error 75 us; skew 500 ppm
I20260809 22:42:19.042918 18786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 22:42:19.045313 18786 webserver.cc:460] Webserver started at http://127.28.243.67:40703/ using document root <none> and password file <none>
I20260809 22:42:19.045964 18786 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:19.046062 18786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:19.046334 18786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:19.049448 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data/instance:
uuid: "259ad04f703c46448799a201557c5827"
format_stamp: "Formatted at 2026-08-09 22:42:19 on dist-test-slave-rgb1"
server_key: "cda618480a4a92ccc175a31d8e66d156"
server_key_iv: "31354955e1bb2656d8bfcb5369c1f137"
server_key_version: "encryptionkey@0"
I20260809 22:42:19.050217 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance:
uuid: "259ad04f703c46448799a201557c5827"
format_stamp: "Formatted at 2026-08-09 22:42:19 on dist-test-slave-rgb1"
server_key: "cda618480a4a92ccc175a31d8e66d156"
server_key_iv: "31354955e1bb2656d8bfcb5369c1f137"
server_key_version: "encryptionkey@0"
I20260809 22:42:19.055696 18786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 22:42:19.059433 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:19.061066 18786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:19.061241 18786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal
uuid: "259ad04f703c46448799a201557c5827"
format_stamp: "Formatted at 2026-08-09 22:42:19 on dist-test-slave-rgb1"
server_key: "cda618480a4a92ccc175a31d8e66d156"
server_key_iv: "31354955e1bb2656d8bfcb5369c1f137"
server_key_version: "encryptionkey@0"
I20260809 22:42:19.061450 18786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:19.075721 18786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:19.076654 18786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:19.076969 18786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:19.078011 18786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 22:42:19.079871 18786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 22:42:19.079980 18786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:19.080098 18786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 22:42:19.080163 18786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:19.117115 18786 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.67:45659
I20260809 22:42:19.117234 18925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.67:45659 every 8 connection(s)
I20260809 22:42:19.118443 18786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/data/info.pb
I20260809 22:42:19.123405 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 18786
I20260809 22:42:19.123636 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal/instance
I20260809 22:42:19.124120 29645 external_mini_cluster.cc:1428] Setting key e78c32622060b8e6eb5f8937a44cfb7c
I20260809 22:42:19.133057 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.121371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.67:34429 (local address 127.28.243.126:33391)
0809 22:42:19.121605 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.121609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.123244 (+ 1635us) server_negotiation.cc:407] Connection header received
0809 22:42:19.125693 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.125698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.125801 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.125919 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:19.127202 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.128060 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.129764 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.130195 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.130404 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.132113 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:19.132119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:19.132122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:19.132144 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:19.132176 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:19.132189 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:19.132297 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.132592 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.132709 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 22:42:19.134274 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.121957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:33391 (local address 127.28.243.67:34429)
0809 22:42:19.123053 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:19.123093 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:19.125342 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:19.126233 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:19.126242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:19.126353 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:42:19.126982 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.127002 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.128221 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.128227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.129611 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.129624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.130494 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.130504 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.130792 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.131641 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 22:42:19.131644 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:42:19.131697 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 22:42:19.131785 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:42:19.132298 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:42:19.132313 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:42:19.132323 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 22:42:19.132554 (+ 231us) client_negotiation.cc:240] Negotiation successful
0809 22:42:19.133050 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":344,"threads_started":1}
I20260809 22:42:19.136245 18926 heartbeater.cc:347] Connected to a master server at 127.28.243.126:33391
I20260809 22:42:19.136695 18926 heartbeater.cc:464] Registering TS with master...
I20260809 22:42:19.137729 18926 heartbeater.cc:511] Master 127.28.243.126:33391 requested a full tablet report, sending...
I20260809 22:42:19.139182 18640 heartbeater.cc:503] Master 127.28.243.126:33391 was elected leader, sending a full tablet report...
I20260809 22:42:19.139495 18436 ts_manager.cc:194] Registered new tserver with Master: 259ad04f703c46448799a201557c5827 (127.28.243.67:45659)
I20260809 22:42:19.140718 29645 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 22:42:19.140729 18436 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.243.67:34429
I20260809 22:42:19.153546 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.146514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.28.243.126:33391)
0809 22:42:19.146789 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.146796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.147157 (+ 361us) server_negotiation.cc:407] Connection header received
0809 22:42:19.147296 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.147301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.147429 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.147565 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:19.148198 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.149158 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.150706 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.151210 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.151373 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.152669 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:19.152675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:19.152678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:19.152700 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:19.152735 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:19.152749 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:19.152863 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.153054 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.153165 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260809 22:42:19.158465 18436 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 22:42:19.166334 18436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 22:42:19.200335 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.181151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:42605 (local address 127.0.0.1:58274)
0809 22:42:19.185433 (+ 4282us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:19.185459 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:19.185634 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:19.186961 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:19.186974 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:19.187018 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:19.187601 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.187619 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.191873 (+ 4254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.191883 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.196634 (+ 4751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.196653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.199190 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.199197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.199468 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.199549 (+ 81us) client_negotiation.cc:769] Sending connection context
0809 22:42:19.199697 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 22:42:19.199915 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2596,"mutex_wait_us":1,"thread_start_us":3036,"threads_started":1}
I20260809 22:42:19.200484 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.181803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.28.243.66:42605)
0809 22:42:19.183424 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.183437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.185802 (+ 2365us) server_negotiation.cc:407] Connection header received
0809 22:42:19.185947 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.185961 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.186350 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.186600 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:19.189395 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.191602 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.196881 (+ 5279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.198730 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.198884 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.199176 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.199849 (+ 673us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.200066 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1428,"thread_start_us":1188,"threads_started":1}
I20260809 22:42:19.204161 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.180923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:45659 (local address 127.0.0.1:42822)
0809 22:42:19.182471 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:19.182526 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:19.182706 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:19.196973 (+ 14267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:19.196979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:19.197006 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:19.197389 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.197400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.199208 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.199212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.201418 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.201433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.203228 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.203235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.203411 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.203459 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 22:42:19.203589 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 22:42:19.203746 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"thread_start_us":1119,"threads_started":1}
I20260809 22:42:19.204210 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.181246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.28.243.67:45659)
0809 22:42:19.196289 (+ 15043us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.196298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.196331 (+ 33us) server_negotiation.cc:407] Connection header received
0809 22:42:19.196425 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.196433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.196742 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.196973 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:19.197579 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.199050 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.201653 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.203000 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.203190 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.203488 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.203632 (+ 144us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.203809 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14864,"thread_start_us":14696,"threads_started":1}
I20260809 22:42:19.205111 18714 tablet_service.cc:1467] Processing CreateTablet for tablet 83492cac1acf4b4a8faac0788080df7c (DEFAULT_TABLE table=test-table [id=3b9833bb8318415db8b0b4037ada5ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:19.206341 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.183695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.65:33639 (local address 127.0.0.1:45878)
0809 22:42:19.195966 (+ 12271us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:19.196002 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:19.196167 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:19.197266 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:19.197276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:19.197311 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:19.197779 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.197793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.199754 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.199763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.202350 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.202368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.205471 (+ 3103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.205478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.205671 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.205705 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 22:42:19.205868 (+ 163us) client_negotiation.cc:240] Negotiation successful
0809 22:42:19.206020 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12080,"thread_start_us":11855,"threads_started":1}
I20260809 22:42:19.206760 18714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83492cac1acf4b4a8faac0788080df7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:19.206902 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.183951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.28.243.65:33639)
0809 22:42:19.195050 (+ 11099us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.195066 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.196256 (+ 1190us) server_negotiation.cc:407] Connection header received
0809 22:42:19.196384 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.196390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.196703 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.196965 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:19.198033 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.199587 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.202586 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.204885 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.205024 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.205337 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.205956 (+ 619us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.206163 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10795,"thread_start_us":11119,"threads_started":1}
I20260809 22:42:19.208515 18852 tablet_service.cc:1467] Processing CreateTablet for tablet 83492cac1acf4b4a8faac0788080df7c (DEFAULT_TABLE table=test-table [id=3b9833bb8318415db8b0b4037ada5ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:19.210438 18852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83492cac1acf4b4a8faac0788080df7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:19.210807 18573 tablet_service.cc:1467] Processing CreateTablet for tablet 83492cac1acf4b4a8faac0788080df7c (DEFAULT_TABLE table=test-table [id=3b9833bb8318415db8b0b4037ada5ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 22:42:19.212214 18573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83492cac1acf4b4a8faac0788080df7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:19.223129 18946 tablet_bootstrap.cc:492] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Bootstrap starting.
I20260809 22:42:19.225023 18947 tablet_bootstrap.cc:492] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Bootstrap starting.
I20260809 22:42:19.227453 18948 tablet_bootstrap.cc:492] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Bootstrap starting.
I20260809 22:42:19.228032 18946 tablet_bootstrap.cc:654] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:19.228718 18947 tablet_bootstrap.cc:654] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:19.229647 18947 log.cc:824] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:19.229652 18946 log.cc:824] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:19.232296 18948 tablet_bootstrap.cc:654] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:19.233220 18946 tablet_bootstrap.cc:492] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: No bootstrap required, opened a new log
I20260809 22:42:19.233210 18947 tablet_bootstrap.cc:492] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: No bootstrap required, opened a new log
I20260809 22:42:19.233491 18946 ts_tablet_manager.cc:1397] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260809 22:42:19.233492 18947 ts_tablet_manager.cc:1397] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260809 22:42:19.233722 18948 log.cc:824] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:19.236603 18948 tablet_bootstrap.cc:492] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: No bootstrap required, opened a new log
I20260809 22:42:19.236912 18948 ts_tablet_manager.cc:1397] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260809 22:42:19.244669 18947 raft_consensus.cc:348] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.244642 18946 raft_consensus.cc:348] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.245151 18947 raft_consensus.cc:374] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:19.245157 18946 raft_consensus.cc:374] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:19.245253 18946 raft_consensus.cc:729] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6dac4c3ee4436aa1d0927cdff5baf2, State: Initialized, Role: FOLLOWER
I20260809 22:42:19.245258 18947 raft_consensus.cc:729] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 259ad04f703c46448799a201557c5827, State: Initialized, Role: FOLLOWER
I20260809 22:42:19.245055 18948 raft_consensus.cc:348] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.245532 18948 raft_consensus.cc:374] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:19.245640 18948 raft_consensus.cc:729] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7effac46ff34d1a954527a5b17da4cf, State: Initialized, Role: FOLLOWER
I20260809 22:42:19.245954 18947 consensus_queue.cc:260] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.246126 18946 consensus_queue.cc:260] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.246356 18948 consensus_queue.cc:260] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.248920 18947 ts_tablet_manager.cc:1428] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Time spent starting tablet: real 0.015s user 0.003s sys 0.012s
I20260809 22:42:19.249265 18948 ts_tablet_manager.cc:1428] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260809 22:42:19.249389 18926 heartbeater.cc:503] Master 127.28.243.126:33391 was elected leader, sending a full tablet report...
I20260809 22:42:19.249466 18946 ts_tablet_manager.cc:1428] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260809 22:42:19.249765 18782 heartbeater.cc:503] Master 127.28.243.126:33391 was elected leader, sending a full tablet report...
I20260809 22:42:19.316975 18954 raft_consensus.cc:482] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 22:42:19.317286 18954 raft_consensus.cc:504] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.319747 18954 leader_election.cc:288] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 259ad04f703c46448799a201557c5827 (127.28.243.67:45659), 0b6dac4c3ee4436aa1d0927cdff5baf2 (127.28.243.66:42605)
I20260809 22:42:19.328398 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.320538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:45985 (local address 127.28.243.66:42605)
0809 22:42:19.320882 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.320893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.321664 (+ 771us) server_negotiation.cc:407] Connection header received
0809 22:42:19.321803 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.321808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.321894 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.321986 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:19.322826 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.323936 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.326644 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.327452 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.327636 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.327712 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.327997 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.328129 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260809 22:42:19.328495 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.320186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.243.65:54401 (local address 127.28.243.67:45659)
0809 22:42:19.320524 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.320531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.321297 (+ 766us) server_negotiation.cc:407] Connection header received
0809 22:42:19.321564 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.321568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.321690 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.321793 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 22:42:19.322802 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.323936 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.326750 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.327598 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.327777 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.327849 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.328124 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.328253 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260809 22:42:19.328521 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.320351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.66:42605 (local address 127.28.243.65:45985)
0809 22:42:19.321534 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:19.321551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:19.321685 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:19.322033 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:19.322040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:19.322084 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:19.322589 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.322604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.324126 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.324132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.326430 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.326444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.327643 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.327649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.327825 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.327850 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 22:42:19.327969 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 22:42:19.328107 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":206,"thread_start_us":767,"threads_started":1}
I20260809 22:42:19.329064 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.320232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.67:45659 (local address 127.28.243.65:54401)
0809 22:42:19.321162 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 22:42:19.321180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 22:42:19.321388 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:42:19.321861 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:42:19.321865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:42:19.321888 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 22:42:19.322502 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.322524 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.324125 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.324132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.326496 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:42:19.326510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.327803 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.327809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:42:19.327958 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.327980 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 22:42:19.328092 (+ 112us) client_negotiation.cc:240] Negotiation successful
0809 22:42:19.328217 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":158,"thread_start_us":526,"threads_started":1}
I20260809 22:42:19.329636 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83492cac1acf4b4a8faac0788080df7c" candidate_uuid: "f7effac46ff34d1a954527a5b17da4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" is_pre_election: true
I20260809 22:42:19.330175 18734 raft_consensus.cc:2454] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7effac46ff34d1a954527a5b17da4cf in term 0.
I20260809 22:42:19.330119 18875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83492cac1acf4b4a8faac0788080df7c" candidate_uuid: "f7effac46ff34d1a954527a5b17da4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259ad04f703c46448799a201557c5827" is_pre_election: true
I20260809 22:42:19.330619 18875 raft_consensus.cc:2454] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7effac46ff34d1a954527a5b17da4cf in term 0.
I20260809 22:42:19.330960 18517 leader_election.cc:302] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6dac4c3ee4436aa1d0927cdff5baf2, f7effac46ff34d1a954527a5b17da4cf; no voters:
I20260809 22:42:19.331395 18954 raft_consensus.cc:2781] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 22:42:19.331558 18954 raft_consensus.cc:482] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 22:42:19.331691 18954 raft_consensus.cc:3037] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:19.334089 18954 raft_consensus.cc:504] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.334739 18954 leader_election.cc:288] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [CANDIDATE]: Term 1 election: Requested vote from peers 259ad04f703c46448799a201557c5827 (127.28.243.67:45659), 0b6dac4c3ee4436aa1d0927cdff5baf2 (127.28.243.66:42605)
I20260809 22:42:19.335284 18875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83492cac1acf4b4a8faac0788080df7c" candidate_uuid: "f7effac46ff34d1a954527a5b17da4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "259ad04f703c46448799a201557c5827"
I20260809 22:42:19.335309 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83492cac1acf4b4a8faac0788080df7c" candidate_uuid: "f7effac46ff34d1a954527a5b17da4cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2"
I20260809 22:42:19.335510 18875 raft_consensus.cc:3037] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:19.335559 18734 raft_consensus.cc:3037] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:19.338382 18875 raft_consensus.cc:2454] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7effac46ff34d1a954527a5b17da4cf in term 1.
I20260809 22:42:19.338618 18734 raft_consensus.cc:2454] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7effac46ff34d1a954527a5b17da4cf in term 1.
I20260809 22:42:19.338864 18515 leader_election.cc:302] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 259ad04f703c46448799a201557c5827, f7effac46ff34d1a954527a5b17da4cf; no voters:
I20260809 22:42:19.339205 18954 raft_consensus.cc:2781] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:19.339723 18954 raft_consensus.cc:686] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 1 LEADER]: Becoming Leader. State: Replica: f7effac46ff34d1a954527a5b17da4cf, State: Running, Role: LEADER
I20260809 22:42:19.340204 18954 consensus_queue.cc:237] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } }
I20260809 22:42:19.344949 18436 catalog_manager.cc:5495] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf reported cstate change: term changed from 0 to 1, leader changed from <none> to f7effac46ff34d1a954527a5b17da4cf (127.28.243.65). New cstate: current_term: 1 leader_uuid: "f7effac46ff34d1a954527a5b17da4cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7effac46ff34d1a954527a5b17da4cf" member_type: VOTER last_known_addr { host: "127.28.243.65" port: 33639 } health_report { overall_health: HEALTHY } } }
W20260809 22:42:19.364688 18642 tablet.cc:2369] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:42:19.370301 18927 tablet.cc:2369] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 22:42:19.375730 18783 tablet.cc:2369] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:42:19.388262 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.382849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.28.243.65:33639)
0809 22:42:19.383183 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.383188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.383208 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:42:19.383305 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.383309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.383413 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.383560 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:42:19.383926 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.384778 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.386216 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.386622 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.386721 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.387191 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:42:19.387707 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:42:19.387827 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.387959 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.388028 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260809 22:42:19.398196 18734 raft_consensus.cc:1261] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 1 FOLLOWER]: Refusing update from remote peer f7effac46ff34d1a954527a5b17da4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:19.399531 18957 consensus_queue.cc:1035] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6dac4c3ee4436aa1d0927cdff5baf2" member_type: VOTER last_known_addr { host: "127.28.243.66" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:19.400309 18875 raft_consensus.cc:1261] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 1 FOLLOWER]: Refusing update from remote peer f7effac46ff34d1a954527a5b17da4cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:42:19.401263 18957 consensus_queue.cc:1035] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "259ad04f703c46448799a201557c5827" member_type: VOTER last_known_addr { host: "127.28.243.67" port: 45659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:42:19.410629 18961 mvcc.cc:205] Tried to move back new op lower bound from 7316747630160699392 to 7316747629941338112. Current Snapshot: MvccSnapshot[applied={T|T < 7316747630160699392}]
I20260809 22:42:19.415587 18962 mvcc.cc:205] Tried to move back new op lower bound from 7316747630160699392 to 7316747629941338112. Current Snapshot: MvccSnapshot[applied={T|T < 7316747630160699392 or (T in {7316747630160699392})}]
I20260809 22:42:19.429776 18435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:19.430197 18435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 22:42:19.434402 18435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9f8cead85fb41e3bbd534035bb913d5: Sending DeleteTablet for 3 replicas of tablet 83492cac1acf4b4a8faac0788080df7c
I20260809 22:42:19.435689 18714 tablet_service.cc:1514] Processing DeleteTablet for tablet 83492cac1acf4b4a8faac0788080df7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:19 UTC) from {username='slave'} at 127.0.0.1:58274
I20260809 22:42:19.435698 18852 tablet_service.cc:1514] Processing DeleteTablet for tablet 83492cac1acf4b4a8faac0788080df7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:19 UTC) from {username='slave'} at 127.0.0.1:42822
I20260809 22:42:19.435698 18573 tablet_service.cc:1514] Processing DeleteTablet for tablet 83492cac1acf4b4a8faac0788080df7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 22:42:19 UTC) from {username='slave'} at 127.0.0.1:45878
I20260809 22:42:19.438292 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 18497
I20260809 22:42:19.448946 18973 tablet_replica.cc:330] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: stopping tablet replica
I20260809 22:42:19.449160 18974 tablet_replica.cc:330] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: stopping tablet replica
I20260809 22:42:19.449512 18972 tablet_replica.cc:330] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: stopping tablet replica
I20260809 22:42:19.449676 18974 raft_consensus.cc:2229] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:19.450182 18973 raft_consensus.cc:2229] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 1 LEADER]: Raft consensus shutting down.
I20260809 22:42:19.450254 18974 raft_consensus.cc:2258] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:19.450627 18972 raft_consensus.cc:2229] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 22:42:19.451215 18972 raft_consensus.cc:2258] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:19.451962 18973 raft_consensus.cc:2258] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 22:42:19.454461 18972 ts_tablet_manager.cc:1905] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:19.454952 18973 ts_tablet_manager.cc:1905] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:19.460906 18974 ts_tablet_manager.cc:1905] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 22:42:19.462356 18972 ts_tablet_manager.cc:1918] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:19.462594 18972 log.cc:1196] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-1/wal/wals/83492cac1acf4b4a8faac0788080df7c
I20260809 22:42:19.463109 18973 ts_tablet_manager.cc:1918] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:19.463325 18973 log.cc:1196] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-0/wal/wals/83492cac1acf4b4a8faac0788080df7c
I20260809 22:42:19.463366 18972 ts_tablet_manager.cc:1939] T 83492cac1acf4b4a8faac0788080df7c P 0b6dac4c3ee4436aa1d0927cdff5baf2: Deleting consensus metadata
I20260809 22:42:19.463948 18973 ts_tablet_manager.cc:1939] T 83492cac1acf4b4a8faac0788080df7c P f7effac46ff34d1a954527a5b17da4cf: Deleting consensus metadata
I20260809 22:42:19.464529 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.440044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.28.243.65:33639)
0809 22:42:19.440233 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.440239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.454058 (+ 13819us) server_negotiation.cc:407] Connection header received
0809 22:42:19.454401 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.454407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.454560 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.454652 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:19.455518 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.456878 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.458920 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.459677 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.459791 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.462486 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:19.462497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:19.462517 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:19.463483 (+ 966us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:19.463565 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:19.463601 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:19.463690 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.463992 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.464114 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:42:19.465986 18635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:19.466014 18422 catalog_manager.cc:4841] TS f7effac46ff34d1a954527a5b17da4cf (127.28.243.65:33639): tablet 83492cac1acf4b4a8faac0788080df7c (table test-table [id=3b9833bb8318415db8b0b4037ada5ab4]) successfully deleted
I20260809 22:42:19.466106 18423 catalog_manager.cc:4841] TS 0b6dac4c3ee4436aa1d0927cdff5baf2 (127.28.243.66:42605): tablet 83492cac1acf4b4a8faac0788080df7c (table test-table [id=3b9833bb8318415db8b0b4037ada5ab4]) successfully deleted
I20260809 22:42:19.472462 18974 ts_tablet_manager.cc:1918] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 22:42:19.472846 18974 log.cc:1196] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Deleting WAL directory at /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786314994965430-29645-0/minicluster-data/ts-2/wal/wals/83492cac1acf4b4a8faac0788080df7c
I20260809 22:42:19.473608 18974 ts_tablet_manager.cc:1939] T 83492cac1acf4b4a8faac0788080df7c P 259ad04f703c46448799a201557c5827: Deleting consensus metadata
I20260809 22:42:19.475476 18421 catalog_manager.cc:4841] TS 259ad04f703c46448799a201557c5827 (127.28.243.67:45659): tablet 83492cac1acf4b4a8faac0788080df7c (table test-table [id=3b9833bb8318415db8b0b4037ada5ab4]) successfully deleted
I20260809 22:42:19.570664 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 18497
I20260809 22:42:19.591773 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 18645
I20260809 22:42:19.599421 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.592723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.28.243.66:42605)
0809 22:42:19.592958 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.592973 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.592996 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:42:19.593248 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.593253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.593349 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.593455 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:19.593878 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.594693 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.596131 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.596605 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.596751 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.597842 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:19.597853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:19.597868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:19.598476 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:19.598523 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:19.598541 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:19.598636 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.598821 (+ 185us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.599047 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 22:42:19.600188 18777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:19.691726 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 18645
I20260809 22:42:19.711030 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 18786
I20260809 22:42:19.718178 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:19.711873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.28.243.67:45659)
0809 22:42:19.712064 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:19.712069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:19.712089 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:42:19.712173 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:19.712177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:19.712274 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:19.712365 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:19.712907 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.713672 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.715131 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:19.715606 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:19.715733 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:19.716890 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:19.716899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:19.716914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:19.717498 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:19.717565 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:19.717584 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:19.717681 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:19.717834 (+ 153us) server_negotiation.cc:299] Negotiation successful
0809 22:42:19.717924 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 22:42:19.718844 18921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:19.812806 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 18786
I20260809 22:42:19.832257 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 18402
I20260809 22:42:19.833607 18467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:19.946866 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 18402
2026-08-09T22:42:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3036 ms)
[----------] 6 tests from EncryptionPolicyTest (24091 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:20 dist-test-slave-rgb1 krb5kdc[18983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:20 dist-test-slave-rgb1 krb5kdc[18983](info): set up 2 sockets
Aug 09 22:42:20 dist-test-slave-rgb1 krb5kdc[18983](info): commencing operation
krb5kdc: starting...
W20260809 22:42:21.213039 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:21 dist-test-slave-rgb1 krb5kdc[18983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:21.578378 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:21.947072 18997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:21.947435 18997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:21.947634 18997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:21.955648 18997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:21.955786 18997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:21.955865 18997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:21.955967 18997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:21.956012 18997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:21.966600 18997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:21.968078 18997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:21.969558 18997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:21.982430 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:21.982543 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:21.982430 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:21.983604 18997 server_base.cc:1034] running on GCE node
I20260809 22:42:21.984728 18997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:21.986377 18997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:21.987674 18997 hybrid_clock.cc:648] HybridClock initialized: now 1786315341987641 us; error 68 us; skew 500 ppm
I20260809 22:42:21.988124 18997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:21 dist-test-slave-rgb1 krb5kdc[18983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:21.997063 18997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:21.999033 18997 webserver.cc:460] Webserver started at http://127.0.0.1:34937/ using document root <none> and password file <none>
I20260809 22:42:21.999822 18997 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:22.000005 18997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:22.000367 18997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:22.003758 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "8998340d43c74a4f88db116a3a4e0f21"
format_stamp: "Formatted at 2026-08-09 22:42:22 on dist-test-slave-rgb1"
server_key: "9b8b39ab4e0a1d4ead9feb9562ffc37d"
server_key_iv: "19ea9a979e7df8e7fc7d79cd57f21eb6"
server_key_version: "encryptionkey@0"
I20260809 22:42:22.004676 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "8998340d43c74a4f88db116a3a4e0f21"
format_stamp: "Formatted at 2026-08-09 22:42:22 on dist-test-slave-rgb1"
server_key: "9b8b39ab4e0a1d4ead9feb9562ffc37d"
server_key_iv: "19ea9a979e7df8e7fc7d79cd57f21eb6"
server_key_version: "encryptionkey@0"
I20260809 22:42:22.011286 18997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:42:22.015586 19014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:22.017441 18997 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:42:22.017710 18997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "8998340d43c74a4f88db116a3a4e0f21"
format_stamp: "Formatted at 2026-08-09 22:42:22 on dist-test-slave-rgb1"
server_key: "9b8b39ab4e0a1d4ead9feb9562ffc37d"
server_key_iv: "19ea9a979e7df8e7fc7d79cd57f21eb6"
server_key_version: "encryptionkey@0"
I20260809 22:42:22.017976 18997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:22.045691 18997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:22.050813 18997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:22.051165 18997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:22.074174 18997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32979
I20260809 22:42:22.074242 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32979 every 8 connection(s)
I20260809 22:42:22.075817 18997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:22.079873 19070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:22.085310 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 18997
I20260809 22:42:22.085568 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:22.086391 29645 external_mini_cluster.cc:1428] Setting key b1a113816420376487b5c1bf48d5e957
I20260809 22:42:22.092100 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21: Bootstrap starting.
I20260809 22:42:22.096117 19070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:22.097479 19070 log.cc:824] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:22.101349 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21: No bootstrap required, opened a new log
Aug 09 22:42:22 dist-test-slave-rgb1 krb5kdc[18983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 22:42:22.110762 19070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998340d43c74a4f88db116a3a4e0f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260809 22:42:22.111178 19070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:22.111315 19070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8998340d43c74a4f88db116a3a4e0f21, State: Initialized, Role: FOLLOWER
I20260809 22:42:22.111990 19070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998340d43c74a4f88db116a3a4e0f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260809 22:42:22.112284 19070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:22.112450 19070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:22.112700 19070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:22.114884 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:22.088749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46581 (local address 127.0.0.1:32979)
0809 22:42:22.089925 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:22.089942 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:22.089994 (+ 52us) server_negotiation.cc:407] Connection header received
0809 22:42:22.091915 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:22.091941 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:22.092468 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:22.092936 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:22.094974 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:22.096684 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:22.098632 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:22.099479 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:22.099626 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:22.105375 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:22.105394 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:22.105413 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:22.105465 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:22.109464 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:22.110601 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:22.110609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:22.110617 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:22.110831 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:22.111487 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:22.111507 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:22.111510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:22.112286 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:22.112726 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:22.113292 (+ 566us) server_negotiation.cc:299] Negotiation successful
0809 22:42:22.113711 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":601,"threads_started":1}
I20260809 22:42:22.115710 19070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998340d43c74a4f88db116a3a4e0f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260809 22:42:22.116328 19070 leader_election.cc:302] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8998340d43c74a4f88db116a3a4e0f21; no voters:
I20260809 22:42:22.117122 19070 leader_election.cc:288] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:22.117372 19076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:22.119673 19076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [term 1 LEADER]: Becoming Leader. State: Replica: 8998340d43c74a4f88db116a3a4e0f21, State: Running, Role: LEADER
I20260809 22:42:22.120450 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:22.120482 19070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:22.120483 19076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998340d43c74a4f88db116a3a4e0f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } }
I20260809 22:42:22.124280 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8998340d43c74a4f88db116a3a4e0f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998340d43c74a4f88db116a3a4e0f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } } }
I20260809 22:42:22.124878 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:22.126904 19078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8998340d43c74a4f88db116a3a4e0f21. Latest consensus state: current_term: 1 leader_uuid: "8998340d43c74a4f88db116a3a4e0f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8998340d43c74a4f88db116a3a4e0f21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32979 } } }
I20260809 22:42:22.127941 19078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:22.133610 19084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:22.141170 19084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:22.163198 19084 catalog_manager.cc:1347] Generated new cluster ID: 37513fa22a914cf5a2816eaa9e23d400
I20260809 22:42:22.163344 19084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:22.179421 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:22.144370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38261 (local address 127.0.0.1:32979)
0809 22:42:22.144654 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:22.144663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:22.152175 (+ 7512us) server_negotiation.cc:407] Connection header received
0809 22:42:22.152537 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:22.152545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:22.152676 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:22.152758 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:22.155099 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:22.156445 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:22.158472 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:22.159175 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:22.159309 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:22.166268 (+ 6959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:22.166280 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:22.166288 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:22.166351 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:22.171547 (+ 5196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:22.173049 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:22.173058 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:22.173064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:22.173294 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:22.174292 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:22.174302 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:22.174308 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:22.174921 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:22.175270 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:22.178499 (+ 3229us) server_negotiation.cc:299] Negotiation successful
0809 22:42:22.178801 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 22:42:22.238399 19084 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:22.240105 19084 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:22.250258 19084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8998340d43c74a4f88db116a3a4e0f21: Generated new TSK 0
I20260809 22:42:22.251241 19084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:22.259801 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 18997
I20260809 22:42:22.261750 19065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:22.365487 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 18997
W20260809 22:42:22.388705 18980 connection.cc:535] client connection to 127.0.0.1:32979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32979): Connection reset by peer (error 104)
2026-08-09T22:42:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:22 dist-test-slave-rgb1 krb5kdc[19106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:22 dist-test-slave-rgb1 krb5kdc[19106](info): set up 2 sockets
Aug 09 22:42:22 dist-test-slave-rgb1 krb5kdc[19106](info): commencing operation
krb5kdc: starting...
W20260809 22:42:23.574575 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:23 dist-test-slave-rgb1 krb5kdc[19106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:23.925318 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:24.287319 19120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:24.287648 19120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:24.287835 19120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:24.295513 19120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:24.295647 19120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:24.295783 19120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:24.295892 19120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:24.295956 19120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:24.306905 19120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:24.308369 19120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:24.309980 19120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:24.323680 19125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:24.323745 19126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:24.323750 19128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:24.325140 19120 server_base.cc:1034] running on GCE node
I20260809 22:42:24.326072 19120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:24.327819 19120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:24.329293 19120 hybrid_clock.cc:648] HybridClock initialized: now 1786315344329185 us; error 98 us; skew 500 ppm
I20260809 22:42:24.329722 19120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:24 dist-test-slave-rgb1 krb5kdc[19106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:24.333442 19120 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:24.335215 19120 webserver.cc:460] Webserver started at http://127.0.0.1:46267/ using document root <none> and password file <none>
I20260809 22:42:24.335968 19120 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:24.336141 19120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:24.336483 19120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:24.340297 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "c0100a4b182a45b0b6cb3f75b968f071"
format_stamp: "Formatted at 2026-08-09 22:42:24 on dist-test-slave-rgb1"
server_key: "0454c411f3f473fd7c44872686997bdf"
server_key_iv: "853b240efbded313e3e4cd63fd10dab8"
server_key_version: "encryptionkey@0"
I20260809 22:42:24.341266 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "c0100a4b182a45b0b6cb3f75b968f071"
format_stamp: "Formatted at 2026-08-09 22:42:24 on dist-test-slave-rgb1"
server_key: "0454c411f3f473fd7c44872686997bdf"
server_key_iv: "853b240efbded313e3e4cd63fd10dab8"
server_key_version: "encryptionkey@0"
I20260809 22:42:24.347203 19120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:42:24.351437 19135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:24.353130 19120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260809 22:42:24.353392 19120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "c0100a4b182a45b0b6cb3f75b968f071"
format_stamp: "Formatted at 2026-08-09 22:42:24 on dist-test-slave-rgb1"
server_key: "0454c411f3f473fd7c44872686997bdf"
server_key_iv: "853b240efbded313e3e4cd63fd10dab8"
server_key_version: "encryptionkey@0"
I20260809 22:42:24.353643 19120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:24.388785 19120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:24.394119 19120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:24.394484 19120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:24.416613 19120 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34113
I20260809 22:42:24.416621 19188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34113 every 8 connection(s)
I20260809 22:42:24.418220 19120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:24.421433 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19120
I20260809 22:42:24.421655 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:24.422224 29645 external_mini_cluster.cc:1428] Setting key 2e7eee3bd9de59d7566ead0cacb351f5
I20260809 22:42:24.423007 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:24.439137 19189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071: Bootstrap starting.
I20260809 22:42:24.439298 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:24.424628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37667 (local address 127.0.0.1:34113)
0809 22:42:24.425628 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:24.425642 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:24.425710 (+ 68us) server_negotiation.cc:407] Connection header received
0809 22:42:24.428068 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:24.428096 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:24.428695 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:24.429258 (+ 563us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:24.430157 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:24.431759 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:24.433655 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:24.434954 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:24.435120 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:24.436654 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:24.436663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:24.436680 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:24.437337 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:24.437420 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:24.437447 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:24.437587 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:24.437975 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 22:42:24.438237 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":419,"threads_started":1}
I20260809 22:42:24.443440 19189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:24.444164 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:24.445101 19189 log.cc:824] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:24.449304 19189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071: No bootstrap required, opened a new log
I20260809 22:42:24.458758 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:24.450196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44999 (local address 127.0.0.1:34113)
0809 22:42:24.450420 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:24.450425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:24.450754 (+ 329us) server_negotiation.cc:407] Connection header received
0809 22:42:24.451006 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:24.451012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:24.451093 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:24.451188 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:24.451925 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:24.452863 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:24.455158 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:24.455786 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:24.455998 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:24.457516 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:24.457528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:24.457541 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:24.457565 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:24.457639 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:24.457661 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:24.457793 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:24.458068 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 22:42:24.458159 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 22:42:24.461314 19189 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100a4b182a45b0b6cb3f75b968f071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260809 22:42:24.461863 19189 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:24.461992 19189 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0100a4b182a45b0b6cb3f75b968f071, State: Initialized, Role: FOLLOWER
I20260809 22:42:24.462702 19189 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100a4b182a45b0b6cb3f75b968f071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260809 22:42:24.463049 19189 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:24.463220 19189 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:24.463438 19189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:24.466055 19189 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100a4b182a45b0b6cb3f75b968f071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260809 22:42:24.466777 19189 leader_election.cc:302] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0100a4b182a45b0b6cb3f75b968f071; no voters:
I20260809 22:42:24.467497 19189 leader_election.cc:288] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:24.467759 19199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:24.468531 19199 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [term 1 LEADER]: Becoming Leader. State: Replica: c0100a4b182a45b0b6cb3f75b968f071, State: Running, Role: LEADER
I20260809 22:42:24.469182 19199 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100a4b182a45b0b6cb3f75b968f071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260809 22:42:24.469877 19189 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:24.472136 19201 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0100a4b182a45b0b6cb3f75b968f071. Latest consensus state: current_term: 1 leader_uuid: "c0100a4b182a45b0b6cb3f75b968f071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100a4b182a45b0b6cb3f75b968f071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } } }
I20260809 22:42:24.472533 19201 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:24.477464 19200 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0100a4b182a45b0b6cb3f75b968f071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0100a4b182a45b0b6cb3f75b968f071" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } } }
I20260809 22:42:24.478615 19200 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:24.480031 19204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:24.485798 19204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:24.496587 19204 catalog_manager.cc:1347] Generated new cluster ID: 25ba2577a9394e84b8388ac28d83f5db
I20260809 22:42:24.496734 19204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:24.521497 19204 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:24.522979 19204 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:24.537559 19204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0100a4b182a45b0b6cb3f75b968f071: Generated new TSK 0
I20260809 22:42:24.538656 19204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:24.545697 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19120
W20260809 22:42:24.546977 19184 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37667
W20260809 22:42:24.548122 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19120: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:24.548393 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19120
W20260809 22:42:24.569572 19103 connection.cc:535] client connection to 127.0.0.1:34113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34113): Connection reset by peer (error 104)
2026-08-09T22:42:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2180 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:24 dist-test-slave-rgb1 krb5kdc[19222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:24 dist-test-slave-rgb1 krb5kdc[19222](info): set up 2 sockets
Aug 09 22:42:24 dist-test-slave-rgb1 krb5kdc[19222](info): commencing operation
krb5kdc: starting...
W20260809 22:42:25.782224 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:26 dist-test-slave-rgb1 krb5kdc[19222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:26.132697 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:26.515297 19236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:26.515609 19236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:26.515704 19236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:26.523936 19236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:26.524091 19236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:26.524144 19236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:26.524206 19236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:26.524258 19236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:26.535662 19236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:26.537164 19236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:26.538487 19236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:26.550958 19244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:26.551050 19242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:26.550906 19241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:26.552196 19236 server_base.cc:1034] running on GCE node
I20260809 22:42:26.553148 19236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:26.554706 19236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:26.556015 19236 hybrid_clock.cc:648] HybridClock initialized: now 1786315346555963 us; error 43 us; skew 500 ppm
I20260809 22:42:26.556408 19236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:26 dist-test-slave-rgb1 krb5kdc[19222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:26.559985 19236 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:26.561623 19236 webserver.cc:460] Webserver started at http://127.0.0.1:37419/ using document root <none> and password file <none>
I20260809 22:42:26.562282 19236 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:26.562390 19236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:26.562638 19236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:26.566260 19236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "305105ecad1c4ac0908df0a8917ad1c7"
format_stamp: "Formatted at 2026-08-09 22:42:26 on dist-test-slave-rgb1"
server_key: "3f2a06d535a12b7ea10260f91a9e851b"
server_key_iv: "d6de3c21e7ae32a31175f98605380add"
server_key_version: "encryptionkey@0"
I20260809 22:42:26.567126 19236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "305105ecad1c4ac0908df0a8917ad1c7"
format_stamp: "Formatted at 2026-08-09 22:42:26 on dist-test-slave-rgb1"
server_key: "3f2a06d535a12b7ea10260f91a9e851b"
server_key_iv: "d6de3c21e7ae32a31175f98605380add"
server_key_version: "encryptionkey@0"
I20260809 22:42:26.572722 19236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 22:42:26.576450 19252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:26.577975 19236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:26.578228 19236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "305105ecad1c4ac0908df0a8917ad1c7"
format_stamp: "Formatted at 2026-08-09 22:42:26 on dist-test-slave-rgb1"
server_key: "3f2a06d535a12b7ea10260f91a9e851b"
server_key_iv: "d6de3c21e7ae32a31175f98605380add"
server_key_version: "encryptionkey@0"
I20260809 22:42:26.578449 19236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:26.584939 19236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:26.590628 19236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:26.591017 19236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:26.613131 19236 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41469
I20260809 22:42:26.613178 19305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41469 every 8 connection(s)
I20260809 22:42:26.614645 19236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:26.618074 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19236
I20260809 22:42:26.618402 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:26.618579 19306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:26.618911 29645 external_mini_cluster.cc:1428] Setting key 15002cff1f8b01548b284ad330b4af31
I20260809 22:42:26.627740 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:26.620830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56103 (local address 127.0.0.1:41469)
0809 22:42:26.621848 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:26.621868 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:26.621924 (+ 56us) server_negotiation.cc:407] Connection header received
0809 22:42:26.623692 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:26.623722 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:26.624190 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:26.624686 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:26.625736 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:26.625745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:26.625761 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:26.625785 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:26.625859 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:26.625892 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:26.626141 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:26.626275 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 22:42:26.626504 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":470,"threads_started":1}
I20260809 22:42:26.631860 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:26.633961 19306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7: Bootstrap starting.
I20260809 22:42:26.638403 19306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:26.639930 19306 log.cc:824] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:26.640658 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:26.637477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54997 (local address 127.0.0.1:41469)
0809 22:42:26.637719 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:26.637729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:26.638243 (+ 514us) server_negotiation.cc:407] Connection header received
0809 22:42:26.638462 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:26.638466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:26.638558 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:26.638693 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:26.639785 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:26.639790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:26.639793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:26.639808 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:26.639847 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:26.639862 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:26.640007 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:26.640209 (+ 202us) server_negotiation.cc:299] Negotiation successful
0809 22:42:26.640297 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 22:42:26.643143 19306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7: No bootstrap required, opened a new log
I20260809 22:42:26.650754 19306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305105ecad1c4ac0908df0a8917ad1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260809 22:42:26.651163 19306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:26.651298 19306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305105ecad1c4ac0908df0a8917ad1c7, State: Initialized, Role: FOLLOWER
I20260809 22:42:26.651903 19306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305105ecad1c4ac0908df0a8917ad1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260809 22:42:26.652151 19306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:26.652330 19306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:26.652510 19306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:26.654893 19306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305105ecad1c4ac0908df0a8917ad1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260809 22:42:26.655419 19306 leader_election.cc:302] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 305105ecad1c4ac0908df0a8917ad1c7; no voters:
I20260809 22:42:26.656098 19306 leader_election.cc:288] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:26.656245 19316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:26.657014 19316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [term 1 LEADER]: Becoming Leader. State: Replica: 305105ecad1c4ac0908df0a8917ad1c7, State: Running, Role: LEADER
I20260809 22:42:26.657761 19316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305105ecad1c4ac0908df0a8917ad1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } }
I20260809 22:42:26.658682 19306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:26.661121 19318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 305105ecad1c4ac0908df0a8917ad1c7. Latest consensus state: current_term: 1 leader_uuid: "305105ecad1c4ac0908df0a8917ad1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305105ecad1c4ac0908df0a8917ad1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } } }
I20260809 22:42:26.661540 19318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:26.661934 19317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "305105ecad1c4ac0908df0a8917ad1c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "305105ecad1c4ac0908df0a8917ad1c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41469 } } }
I20260809 22:42:26.662294 19317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:26.664126 19324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:26.669577 19324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:26.679611 19324 catalog_manager.cc:1347] Generated new cluster ID: 897446b838304f57911f0caf46dd61eb
I20260809 22:42:26.679752 19324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:26.694770 19324 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:26.695230 19324 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:26.711350 19324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 305105ecad1c4ac0908df0a8917ad1c7: Generated new TSK 0
I20260809 22:42:26.712452 19324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:26.719590 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19236
W20260809 22:42:26.721225 19301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56103
W20260809 22:42:26.721956 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19236: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:26.722129 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19236
2026-08-09T22:42:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2170 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:26 dist-test-slave-rgb1 krb5kdc[19338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:26 dist-test-slave-rgb1 krb5kdc[19338](info): set up 2 sockets
Aug 09 22:42:26 dist-test-slave-rgb1 krb5kdc[19338](info): commencing operation
krb5kdc: starting...
W20260809 22:42:27.913301 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.010s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:28 dist-test-slave-rgb1 krb5kdc[19338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:28.266251 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:28.631345 19351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:28.631742 19351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:28.631937 19351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:28.640409 19351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:28.640630 19351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:28.640758 19351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:28.640898 19351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:28.641007 19351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:28.652702 19351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:28.654251 19351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:28.655709 19351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:28.668812 19357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:28.668740 19358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:28.668902 19360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:28.669567 19351 server_base.cc:1034] running on GCE node
I20260809 22:42:28.670403 19351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:28.671988 19351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:28.673229 19351 hybrid_clock.cc:648] HybridClock initialized: now 1786315348673175 us; error 48 us; skew 500 ppm
I20260809 22:42:28.673626 19351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:28 dist-test-slave-rgb1 krb5kdc[19338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:28.682420 19351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:28.684269 19351 webserver.cc:460] Webserver started at http://127.0.0.1:44419/ using document root <none> and password file <none>
I20260809 22:42:28.685067 19351 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:28.685227 19351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:28.685561 19351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:28.688792 19351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "8f69fdd374524b0e9c9ea70a4d671e39"
format_stamp: "Formatted at 2026-08-09 22:42:28 on dist-test-slave-rgb1"
server_key: "9638d5f2597a8f037ef006a694b7081a"
server_key_iv: "7e21ab7ac114802d2292f8f7ee680771"
server_key_version: "encryptionkey@0"
I20260809 22:42:28.689625 19351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "8f69fdd374524b0e9c9ea70a4d671e39"
format_stamp: "Formatted at 2026-08-09 22:42:28 on dist-test-slave-rgb1"
server_key: "9638d5f2597a8f037ef006a694b7081a"
server_key_iv: "7e21ab7ac114802d2292f8f7ee680771"
server_key_version: "encryptionkey@0"
I20260809 22:42:28.695220 19351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:42:28.699090 19367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:28.700686 19351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 22:42:28.700886 19351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "8f69fdd374524b0e9c9ea70a4d671e39"
format_stamp: "Formatted at 2026-08-09 22:42:28 on dist-test-slave-rgb1"
server_key: "9638d5f2597a8f037ef006a694b7081a"
server_key_iv: "7e21ab7ac114802d2292f8f7ee680771"
server_key_version: "encryptionkey@0"
I20260809 22:42:28.701123 19351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:28.731117 19351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:28.736057 19351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:28.736352 19351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:28.756970 19351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42759
I20260809 22:42:28.757004 19423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42759 every 8 connection(s)
I20260809 22:42:28.758422 19351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:28.762884 19424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:28.762907 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19351
I20260809 22:42:28.763128 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:28.763635 29645 external_mini_cluster.cc:1428] Setting key bc12ffd87350a52954da2c8cbe9d2230
I20260809 22:42:28.778496 19424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39: Bootstrap starting.
Aug 09 22:42:28 dist-test-slave-rgb1 krb5kdc[19338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 22:42:28.782267 19424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:28.783416 19424 log.cc:824] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:28.787411 19424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39: No bootstrap required, opened a new log
I20260809 22:42:28.790360 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:28.765465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38861 (local address 127.0.0.1:42759)
0809 22:42:28.766325 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:28.766338 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:28.766384 (+ 46us) server_negotiation.cc:407] Connection header received
0809 22:42:28.768189 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:28.768211 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:28.768689 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:28.769059 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:28.771135 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:28.772496 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:28.774426 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:28.775157 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:28.775313 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:28.780949 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:28.780960 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:28.780976 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:28.781021 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:28.785306 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:28.786497 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:28.786504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:28.786510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:28.786703 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:28.787213 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:28.787217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:28.787220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:28.787849 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:28.788120 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:28.788846 (+ 726us) server_negotiation.cc:299] Negotiation successful
0809 22:42:28.789281 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":401,"threads_started":1}
I20260809 22:42:28.794387 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:28.797108 19424 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260809 22:42:28.797601 19424 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:28.797729 19424 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f69fdd374524b0e9c9ea70a4d671e39, State: Initialized, Role: FOLLOWER
I20260809 22:42:28.798473 19424 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260809 22:42:28.798753 19424 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:28.798925 19424 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:28.799135 19424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:28.803138 19424 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260809 22:42:28.803761 19424 leader_election.cc:302] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f69fdd374524b0e9c9ea70a4d671e39; no voters:
I20260809 22:42:28.804692 19424 leader_election.cc:288] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:28.804850 19434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:28.805791 19434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [term 1 LEADER]: Becoming Leader. State: Replica: 8f69fdd374524b0e9c9ea70a4d671e39, State: Running, Role: LEADER
I20260809 22:42:28.806491 19434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } }
I20260809 22:42:28.810061 19436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } } }
I20260809 22:42:28.810541 19436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:28.811503 19435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f69fdd374524b0e9c9ea70a4d671e39. Latest consensus state: current_term: 1 leader_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f69fdd374524b0e9c9ea70a4d671e39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42759 } } }
I20260809 22:42:28.811913 19435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:28.814429 19424 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:28.821815 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:28.799572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49559 (local address 127.0.0.1:42759)
0809 22:42:28.799840 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:28.799851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:28.800279 (+ 428us) server_negotiation.cc:407] Connection header received
0809 22:42:28.800457 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:28.800461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:28.800537 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:28.800697 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:28.802918 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:28.803774 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:28.806475 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:28.807198 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:28.807340 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:28.811289 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:28.811302 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:28.811310 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:28.811377 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:28.816460 (+ 5083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:28.818180 (+ 1720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:28.818189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:28.818196 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:28.818396 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:28.819101 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:28.819139 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:28.819145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:28.819810 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:28.820134 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:28.821000 (+ 866us) server_negotiation.cc:299] Negotiation successful
0809 22:42:28.821344 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:42:28.822077 19438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:28.829263 19438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:28.838816 19438 catalog_manager.cc:1347] Generated new cluster ID: 9379e0b4e535490d9c27b1faccfcd822
I20260809 22:42:28.839068 19438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:28.856221 19438 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:28.857764 19438 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:28.865837 19438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f69fdd374524b0e9c9ea70a4d671e39: Generated new TSK 0
I20260809 22:42:28.866803 19438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:28.877357 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19351
I20260809 22:42:28.878888 19419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:28.972581 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19351
2026-08-09T22:42:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:29 dist-test-slave-rgb1 krb5kdc[19458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:29 dist-test-slave-rgb1 krb5kdc[19458](info): set up 2 sockets
Aug 09 22:42:29 dist-test-slave-rgb1 krb5kdc[19458](info): commencing operation
krb5kdc: starting...
W20260809 22:42:30.223277 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.003s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:30 dist-test-slave-rgb1 krb5kdc[19458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:30.596983 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:30.991436 19473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:30.991833 19473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:30.992092 19473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:31.001109 19473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:31.001262 19473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:31.001350 19473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:31.001452 19473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:31.001493 19473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:31.012212 19473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:31.013746 19473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:31.015072 19473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:31.028205 19479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:31.028237 19480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:31.028265 19482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:31.029315 19473 server_base.cc:1034] running on GCE node
I20260809 22:42:31.030224 19473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:31.031791 19473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:31.033027 19473 hybrid_clock.cc:648] HybridClock initialized: now 1786315351032979 us; error 37 us; skew 500 ppm
I20260809 22:42:31.033435 19473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:31 dist-test-slave-rgb1 krb5kdc[19458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:31.037214 19473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:31.039103 19473 webserver.cc:460] Webserver started at http://127.0.0.1:36693/ using document root <none> and password file <none>
I20260809 22:42:31.039815 19473 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:31.039984 19473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:31.040390 19473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:31.043622 19473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "6d652cde55f0405091f87e585593a15a"
format_stamp: "Formatted at 2026-08-09 22:42:31 on dist-test-slave-rgb1"
server_key: "0bebc161ef4396db9c01515288e6d7ee"
server_key_iv: "898580a87aa567378bc7f7d63b10572c"
server_key_version: "encryptionkey@0"
I20260809 22:42:31.044461 19473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "6d652cde55f0405091f87e585593a15a"
format_stamp: "Formatted at 2026-08-09 22:42:31 on dist-test-slave-rgb1"
server_key: "0bebc161ef4396db9c01515288e6d7ee"
server_key_iv: "898580a87aa567378bc7f7d63b10572c"
server_key_version: "encryptionkey@0"
I20260809 22:42:31.050129 19473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 22:42:31.054149 19489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:31.055737 19473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:42:31.055935 19473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "6d652cde55f0405091f87e585593a15a"
format_stamp: "Formatted at 2026-08-09 22:42:31 on dist-test-slave-rgb1"
server_key: "0bebc161ef4396db9c01515288e6d7ee"
server_key_iv: "898580a87aa567378bc7f7d63b10572c"
server_key_version: "encryptionkey@0"
I20260809 22:42:31.056146 19473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:31.074023 19473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:31.079062 19473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:31.079401 19473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:31.100625 19473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42851
I20260809 22:42:31.100720 19543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42851 every 8 connection(s)
I20260809 22:42:31.102088 19473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:31.105085 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19473
I20260809 22:42:31.105320 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:31.105935 29645 external_mini_cluster.cc:1428] Setting key 21c1eb4bc569bcf1b62b7b78a2ccfdc4
I20260809 22:42:31.107050 19544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:31.123603 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:31.107893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51567 (local address 127.0.0.1:42851)
0809 22:42:31.108854 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:31.108872 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:31.108934 (+ 62us) server_negotiation.cc:407] Connection header received
0809 22:42:31.111243 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:31.111278 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:31.111954 (+ 676us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:31.112527 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:31.113219 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:31.115341 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:31.117106 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:31.118211 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:31.118399 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:31.119781 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:31.119792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:31.119818 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:31.120652 (+ 834us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:31.120767 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:31.120800 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:31.120923 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:31.121579 (+ 656us) server_negotiation.cc:299] Negotiation successful
0809 22:42:31.121949 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":389,"threads_started":1}
I20260809 22:42:31.124297 19544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a: Bootstrap starting.
I20260809 22:42:31.127873 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:31.128330 19544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:31.129633 19544 log.cc:824] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:31.133749 19544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a: No bootstrap required, opened a new log
I20260809 22:42:31.140825 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:31.132787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47893 (local address 127.0.0.1:42851)
0809 22:42:31.133036 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:31.133044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:31.133511 (+ 467us) server_negotiation.cc:407] Connection header received
0809 22:42:31.133693 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:31.133703 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:31.133820 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:31.133964 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:31.134511 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:31.135778 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:31.137303 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:31.138182 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:31.138384 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:31.139709 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:31.139726 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:31.139733 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:31.139767 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:31.139838 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:31.139869 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:31.140002 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:31.140189 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 22:42:31.140318 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 22:42:31.145076 19544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d652cde55f0405091f87e585593a15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260809 22:42:31.145591 19544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:31.145774 19544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d652cde55f0405091f87e585593a15a, State: Initialized, Role: FOLLOWER
I20260809 22:42:31.146611 19544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d652cde55f0405091f87e585593a15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260809 22:42:31.146974 19544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:31.147174 19544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:31.147432 19544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:31.150202 19544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d652cde55f0405091f87e585593a15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260809 22:42:31.150796 19544 leader_election.cc:302] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d652cde55f0405091f87e585593a15a; no voters:
I20260809 22:42:31.151461 19544 leader_election.cc:288] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:31.151611 19555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:31.152509 19555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [term 1 LEADER]: Becoming Leader. State: Replica: 6d652cde55f0405091f87e585593a15a, State: Running, Role: LEADER
I20260809 22:42:31.153275 19555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d652cde55f0405091f87e585593a15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } }
I20260809 22:42:31.154527 19544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:31.156137 19557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d652cde55f0405091f87e585593a15a. Latest consensus state: current_term: 1 leader_uuid: "6d652cde55f0405091f87e585593a15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d652cde55f0405091f87e585593a15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } } }
I20260809 22:42:31.156693 19557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:31.156787 19556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d652cde55f0405091f87e585593a15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d652cde55f0405091f87e585593a15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42851 } } }
I20260809 22:42:31.157258 19556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:31.165802 19563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:31.171976 19563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:31.181660 19563 catalog_manager.cc:1347] Generated new cluster ID: 826b31a7937745f3930686f49010e90f
I20260809 22:42:31.181820 19563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:31.195329 19563 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:31.197369 19563 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:31.210233 19563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d652cde55f0405091f87e585593a15a: Generated new TSK 0
I20260809 22:42:31.211303 19563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:31.226874 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19473
W20260809 22:42:31.228406 19538 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51567
W20260809 22:42:31.229336 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19473: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:31.229521 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19473
2026-08-09T22:42:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:31 dist-test-slave-rgb1 krb5kdc[19578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:31 dist-test-slave-rgb1 krb5kdc[19578](info): set up 2 sockets
Aug 09 22:42:31 dist-test-slave-rgb1 krb5kdc[19578](info): commencing operation
krb5kdc: starting...
W20260809 22:42:32.513638 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.243s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:32 dist-test-slave-rgb1 krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:32.874305 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:33.282603 19591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:33.282964 19591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:33.283157 19591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:33.291203 19591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:33.291354 19591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:33.291478 19591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:33.291596 19591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:33.291671 19591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:33.302431 19591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:33.303886 19591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:33.305398 19591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:33.319729 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:33.319761 19597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:33.320647 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:33.321635 19591 server_base.cc:1034] running on GCE node
I20260809 22:42:33.322574 19591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:33.324309 19591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:33.325593 19591 hybrid_clock.cc:648] HybridClock initialized: now 1786315353325535 us; error 90 us; skew 500 ppm
I20260809 22:42:33.326071 19591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:33 dist-test-slave-rgb1 krb5kdc[19578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:33.330123 19591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:33.331905 19591 webserver.cc:460] Webserver started at http://127.0.0.1:43177/ using document root <none> and password file <none>
I20260809 22:42:33.332727 19591 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:33.332883 19591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:33.333223 19591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:33.336673 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "ebc96b1d6bf84f148f261c4b7ae22316"
format_stamp: "Formatted at 2026-08-09 22:42:33 on dist-test-slave-rgb1"
server_key: "787ebbd2a2bb7f43c479885edb879289"
server_key_iv: "ab6686c988d8f5abf43cef293d5d19e7"
server_key_version: "encryptionkey@0"
I20260809 22:42:33.337641 19591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "ebc96b1d6bf84f148f261c4b7ae22316"
format_stamp: "Formatted at 2026-08-09 22:42:33 on dist-test-slave-rgb1"
server_key: "787ebbd2a2bb7f43c479885edb879289"
server_key_iv: "ab6686c988d8f5abf43cef293d5d19e7"
server_key_version: "encryptionkey@0"
I20260809 22:42:33.344071 19591 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 22:42:33.348357 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:33.350318 19591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:42:33.350566 19591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "ebc96b1d6bf84f148f261c4b7ae22316"
format_stamp: "Formatted at 2026-08-09 22:42:33 on dist-test-slave-rgb1"
server_key: "787ebbd2a2bb7f43c479885edb879289"
server_key_iv: "ab6686c988d8f5abf43cef293d5d19e7"
server_key_version: "encryptionkey@0"
I20260809 22:42:33.350812 19591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:33.357987 19591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:33.363103 19591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:33.363481 19591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:33.386487 19591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35179
I20260809 22:42:33.386528 19660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35179 every 8 connection(s)
I20260809 22:42:33.388065 19591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:33.392103 19662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:33.392745 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19591
I20260809 22:42:33.392944 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:33.393816 29645 external_mini_cluster.cc:1428] Setting key 525491f888915569ee53a274f1adb8a3
I20260809 22:42:33.406159 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:33.395924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49345 (local address 127.0.0.1:35179)
0809 22:42:33.397295 (+ 1371us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:33.397311 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:33.397392 (+ 81us) server_negotiation.cc:407] Connection header received
0809 22:42:33.401482 (+ 4090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:33.401520 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:33.402189 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:33.402820 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:33.404018 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:33.404029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:33.404053 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:33.404072 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:33.404147 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:33.404172 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:33.404468 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:33.404667 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 22:42:33.404898 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":611,"threads_started":1}
I20260809 22:42:33.410856 19662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316: Bootstrap starting.
I20260809 22:42:33.411098 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:33.415591 19662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:33.417124 19662 log.cc:824] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:33.420517 19665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:33.417584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54179 (local address 127.0.0.1:35179)
0809 22:42:33.417803 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:33.417810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:33.418174 (+ 364us) server_negotiation.cc:407] Connection header received
0809 22:42:33.418361 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:33.418365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:33.418432 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:33.418524 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:33.419689 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:33.419694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:33.419697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:33.419712 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:33.419757 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:33.419773 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:33.419885 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:33.420040 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 22:42:33.420142 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 22:42:33.423791 19662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316: No bootstrap required, opened a new log
I20260809 22:42:33.433080 19662 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260809 22:42:33.433625 19662 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:33.433766 19662 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc96b1d6bf84f148f261c4b7ae22316, State: Initialized, Role: FOLLOWER
I20260809 22:42:33.434541 19662 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260809 22:42:33.434868 19662 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:33.435060 19662 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:33.435274 19662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:33.438416 19662 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260809 22:42:33.439136 19662 leader_election.cc:302] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebc96b1d6bf84f148f261c4b7ae22316; no voters:
I20260809 22:42:33.439832 19662 leader_election.cc:288] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:33.439976 19672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:33.441347 19672 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [term 1 LEADER]: Becoming Leader. State: Replica: ebc96b1d6bf84f148f261c4b7ae22316, State: Running, Role: LEADER
I20260809 22:42:33.441929 19672 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } }
I20260809 22:42:33.442364 19662 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:33.445362 19674 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebc96b1d6bf84f148f261c4b7ae22316. Latest consensus state: current_term: 1 leader_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } } }
I20260809 22:42:33.445837 19674 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:33.446404 19673 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebc96b1d6bf84f148f261c4b7ae22316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35179 } } }
I20260809 22:42:33.446786 19673 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:33.448734 19678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:33.456723 19678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:33.467708 19678 catalog_manager.cc:1347] Generated new cluster ID: e677066b7bad4056907db8f31fe8ed57
I20260809 22:42:33.467846 19678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:33.494333 19678 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:33.494935 19678 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:33.505735 19678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebc96b1d6bf84f148f261c4b7ae22316: Generated new TSK 0
I20260809 22:42:33.506709 19678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:33.524858 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19591
W20260809 22:42:33.526137 19655 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49345
W20260809 22:42:33.527062 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:33.527231 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19591
2026-08-09T22:42:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:33 dist-test-slave-rgb1 krb5kdc[19693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:33 dist-test-slave-rgb1 krb5kdc[19693](info): set up 2 sockets
Aug 09 22:42:33 dist-test-slave-rgb1 krb5kdc[19693](info): commencing operation
krb5kdc: starting...
W20260809 22:42:34.761214 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.008s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:34 dist-test-slave-rgb1 krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:35.129737 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:35.501780 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:35.502220 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:35.502414 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:35.510609 19706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:35.510821 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:35.510938 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:35.511076 19706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:35.511160 19706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:35.521697 19706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:35.523248 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:35.524689 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:35.538214 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:35.538137 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:35.538187 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:35.539675 19706 server_base.cc:1034] running on GCE node
I20260809 22:42:35.540715 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:35.542305 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:35.543597 19706 hybrid_clock.cc:648] HybridClock initialized: now 1786315355543526 us; error 64 us; skew 500 ppm
I20260809 22:42:35.544045 19706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:35 dist-test-slave-rgb1 krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:35.552932 19706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:35.555572 19706 webserver.cc:460] Webserver started at http://127.0.0.1:39851/ using document root <none> and password file <none>
I20260809 22:42:35.556350 19706 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:35.556470 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:35.556833 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:35.560096 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "6e754b806670476288513041a025bf56"
format_stamp: "Formatted at 2026-08-09 22:42:35 on dist-test-slave-rgb1"
server_key: "bdcfbf0da1bc78d52b0e5258782b158a"
server_key_iv: "053fca2304696754ec50cdbd9f97c605"
server_key_version: "encryptionkey@0"
I20260809 22:42:35.560964 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "6e754b806670476288513041a025bf56"
format_stamp: "Formatted at 2026-08-09 22:42:35 on dist-test-slave-rgb1"
server_key: "bdcfbf0da1bc78d52b0e5258782b158a"
server_key_iv: "053fca2304696754ec50cdbd9f97c605"
server_key_version: "encryptionkey@0"
I20260809 22:42:35.568238 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260809 22:42:35.572620 19722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:35.574452 19706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:42:35.574911 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "6e754b806670476288513041a025bf56"
format_stamp: "Formatted at 2026-08-09 22:42:35 on dist-test-slave-rgb1"
server_key: "bdcfbf0da1bc78d52b0e5258782b158a"
server_key_iv: "053fca2304696754ec50cdbd9f97c605"
server_key_version: "encryptionkey@0"
I20260809 22:42:35.575325 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:35.594525 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:35.599699 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:35.599972 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:35.621184 19706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46767
I20260809 22:42:35.621284 19774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46767 every 8 connection(s)
I20260809 22:42:35.622704 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:35.625792 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19706
I20260809 22:42:35.625990 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:35.626523 29645 external_mini_cluster.cc:1428] Setting key 97e595278b9652ff0124787252013fa0
I20260809 22:42:35.627281 19775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:35.640750 19775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56: Bootstrap starting.
Aug 09 22:42:35 dist-test-slave-rgb1 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 22:42:35.644212 19775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:35.645429 19775 log.cc:824] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:35.648847 19775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56: No bootstrap required, opened a new log
I20260809 22:42:35.652525 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:35.628485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:47521 (local address 127.0.0.1:46767)
0809 22:42:35.629490 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:35.629508 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:35.629849 (+ 341us) server_negotiation.cc:407] Connection header received
0809 22:42:35.632832 (+ 2983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:35.632858 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:35.633353 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:35.633800 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:35.635414 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:35.636820 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:35.638127 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:35.638887 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:35.639115 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:35.643966 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:35.643973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:35.643988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:35.644031 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:35.647855 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:35.648854 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:35.648861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:35.648867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:35.649138 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:35.649684 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:35.649688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:35.649690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:35.650218 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:35.650626 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:35.651078 (+ 452us) server_negotiation.cc:299] Negotiation successful
0809 22:42:35.651434 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":439,"threads_started":1}
I20260809 22:42:35.657420 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:35.657995 19775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e754b806670476288513041a025bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260809 22:42:35.658398 19775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:35.658520 19775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e754b806670476288513041a025bf56, State: Initialized, Role: FOLLOWER
I20260809 22:42:35.659178 19775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e754b806670476288513041a025bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260809 22:42:35.659427 19775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:35.659587 19775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:35.659772 19775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:35.662873 19775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e754b806670476288513041a025bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260809 22:42:35.663630 19775 leader_election.cc:302] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e754b806670476288513041a025bf56; no voters:
I20260809 22:42:35.664361 19775 leader_election.cc:288] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:35.664599 19787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:35.665484 19787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [term 1 LEADER]: Becoming Leader. State: Replica: 6e754b806670476288513041a025bf56, State: Running, Role: LEADER
I20260809 22:42:35.666327 19787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e754b806670476288513041a025bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } }
I20260809 22:42:35.668773 19787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e754b806670476288513041a025bf56. Latest consensus state: current_term: 1 leader_uuid: "6e754b806670476288513041a025bf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e754b806670476288513041a025bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } } }
I20260809 22:42:35.669144 19787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:35.669634 19789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e754b806670476288513041a025bf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e754b806670476288513041a025bf56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46767 } } }
I20260809 22:42:35.670184 19789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:35.673818 19775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:35.680823 19791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:35.686520 19791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:35.689249 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:35.663390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:42731 (local address 127.0.0.1:46767)
0809 22:42:35.663624 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:35.663632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:35.664195 (+ 563us) server_negotiation.cc:407] Connection header received
0809 22:42:35.664426 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:35.664432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:35.664525 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:35.664699 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:35.667148 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:35.668829 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:35.671534 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:35.672310 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:35.672488 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:35.677353 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:35.677360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:35.677367 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:35.677426 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:35.682903 (+ 5477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:35.684493 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:35.684505 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:35.684511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:35.684763 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:35.685574 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:35.685582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:35.685588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:35.686171 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:35.688133 (+ 1962us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:35.688376 (+ 243us) server_negotiation.cc:299] Negotiation successful
0809 22:42:35.688682 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 22:42:35.705293 19791 catalog_manager.cc:1347] Generated new cluster ID: 7a4177ef28304a9592474ae5c4233351
I20260809 22:42:35.705507 19791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:35.721968 19791 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:35.723459 19791 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:35.734908 19791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e754b806670476288513041a025bf56: Generated new TSK 0
I20260809 22:42:35.735785 19791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:35.751184 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19706
I20260809 22:42:35.752446 19770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:35.847325 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19706
W20260809 22:42:35.869169 19691 connection.cc:535] client connection to 127.0.0.1:46767 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46767): Connection reset by peer (error 104)
2026-08-09T22:42:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:35 dist-test-slave-rgb1 krb5kdc[19809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:35 dist-test-slave-rgb1 krb5kdc[19809](info): set up 2 sockets
Aug 09 22:42:35 dist-test-slave-rgb1 krb5kdc[19809](info): commencing operation
krb5kdc: starting...
W20260809 22:42:37.075044 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:37 dist-test-slave-rgb1 krb5kdc[19809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:37.437994 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45013
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:37.797338 19822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:37.797641 19822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:37.797792 19822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:37.805287 19822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:37.805399 19822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:37.805485 19822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:37.805589 19822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:37.805655 19822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:37.816453 19822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:37.817860 19822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:37.819199 19822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:37.832279 19827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:37.832316 19828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:37.833300 19831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:37.833252 19822 server_base.cc:1034] running on GCE node
I20260809 22:42:37.834273 19822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:37.835889 19822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:37.837168 19822 hybrid_clock.cc:648] HybridClock initialized: now 1786315357837126 us; error 44 us; skew 500 ppm
I20260809 22:42:37.837633 19822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:37 dist-test-slave-rgb1 krb5kdc[19809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:37.841536 19822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:37.843396 19822 webserver.cc:460] Webserver started at http://127.0.0.1:44355/ using document root <none> and password file <none>
I20260809 22:42:37.844173 19822 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:37.844327 19822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:37.844744 19822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:37.848208 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "443c9dd0178f4c41a9002b27daa3a009"
format_stamp: "Formatted at 2026-08-09 22:42:37 on dist-test-slave-rgb1"
server_key: "fa10621a24747bc637b0df4558830ed4"
server_key_iv: "9ce371b3ca7fcbb8df032e165fbe920f"
server_key_version: "encryptionkey@0"
I20260809 22:42:37.849260 19822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "443c9dd0178f4c41a9002b27daa3a009"
format_stamp: "Formatted at 2026-08-09 22:42:37 on dist-test-slave-rgb1"
server_key: "fa10621a24747bc637b0df4558830ed4"
server_key_iv: "9ce371b3ca7fcbb8df032e165fbe920f"
server_key_version: "encryptionkey@0"
I20260809 22:42:37.855463 19822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260809 22:42:37.859550 19838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:37.861328 19822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:37.861552 19822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "443c9dd0178f4c41a9002b27daa3a009"
format_stamp: "Formatted at 2026-08-09 22:42:37 on dist-test-slave-rgb1"
server_key: "fa10621a24747bc637b0df4558830ed4"
server_key_iv: "9ce371b3ca7fcbb8df032e165fbe920f"
server_key_version: "encryptionkey@0"
I20260809 22:42:37.861783 19822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:37.880875 19822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:37.885675 19822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:37.886008 19822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:37.906419 19822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45013
I20260809 22:42:37.906458 19892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45013 every 8 connection(s)
I20260809 22:42:37.907861 19822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:37.911795 19893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:37.913637 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19822
I20260809 22:42:37.913839 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:37.914304 29645 external_mini_cluster.cc:1428] Setting key d03a48300e5e51ec1d9af56f72a924fe
I20260809 22:42:37.926991 19893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009: Bootstrap starting.
I20260809 22:42:37.930068 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:37.916276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:36165 (local address 127.0.0.1:45013)
0809 22:42:37.917214 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:37.917230 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:37.917283 (+ 53us) server_negotiation.cc:407] Connection header received
0809 22:42:37.919135 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:37.919169 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:37.919763 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:37.920219 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:37.920818 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:37.922768 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:37.924320 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:37.925462 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:37.925686 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:37.927011 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:37.927032 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:37.927053 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:37.927805 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:37.927906 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:37.927940 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:37.928101 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:37.928439 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 22:42:37.928671 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":431,"threads_started":1}
I20260809 22:42:37.931104 19893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:37.932237 19893 log.cc:824] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:37.934638 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:37.935683 19893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009: No bootstrap required, opened a new log
I20260809 22:42:37.946556 19893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443c9dd0178f4c41a9002b27daa3a009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45013 } }
I20260809 22:42:37.947067 19893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:37.947203 19893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443c9dd0178f4c41a9002b27daa3a009, State: Initialized, Role: FOLLOWER
I20260809 22:42:37.947909 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:37.940396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:52963 (local address 127.0.0.1:45013)
0809 22:42:37.940674 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:37.940681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:37.941117 (+ 436us) server_negotiation.cc:407] Connection header received
0809 22:42:37.941294 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:37.941301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:37.941400 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:37.941550 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:37.942095 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:37.943323 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:37.944829 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:37.945554 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:37.945684 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:37.946987 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:37.946994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:37.946999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:37.947036 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:37.947093 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:37.947120 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:37.947261 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:37.947538 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 22:42:37.947597 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 22:42:37.947999 19893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443c9dd0178f4c41a9002b27daa3a009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45013 } }
I20260809 22:42:37.948304 19893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:37.948479 19893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:37.948727 19893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:37.951114 19893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443c9dd0178f4c41a9002b27daa3a009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45013 } }
I20260809 22:42:37.951709 19893 leader_election.cc:302] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443c9dd0178f4c41a9002b27daa3a009; no voters:
I20260809 22:42:37.952368 19893 leader_election.cc:288] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:37.952597 19904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:37.953485 19904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [term 1 LEADER]: Becoming Leader. State: Replica: 443c9dd0178f4c41a9002b27daa3a009, State: Running, Role: LEADER
I20260809 22:42:37.954118 19904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443c9dd0178f4c41a9002b27daa3a009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45013 } }
I20260809 22:42:37.954660 19893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:37.958976 19906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443c9dd0178f4c41a9002b27daa3a009. Latest consensus state: current_term: 1 leader_uuid: "443c9dd0178f4c41a9002b27daa3a009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443c9dd0178f4c41a9002b27daa3a009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45013 } } }
I20260809 22:42:37.958827 19905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "443c9dd0178f4c41a9002b27daa3a009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443c9dd0178f4c41a9002b27daa3a009" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45013 } } }
I20260809 22:42:37.959474 19906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:37.959600 19905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:37.961282 19914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:37.967110 19914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:37.978514 19914 catalog_manager.cc:1347] Generated new cluster ID: 82f640e492594d9f97a1eb7b55d3502a
I20260809 22:42:37.978708 19914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:37.996752 19914 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:37.998405 19914 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:38.007072 19914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443c9dd0178f4c41a9002b27daa3a009: Generated new TSK 0
I20260809 22:42:38.008026 19914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:38.022915 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19822
W20260809 22:42:38.024784 19888 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.19:36165
W20260809 22:42:38.025691 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19822: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:38.025862 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19822
W20260809 22:42:38.047473 19807 connection.cc:535] client connection to 127.0.0.1:45013 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45013): Connection reset by peer (error 104)
2026-08-09T22:42:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2177 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:38 dist-test-slave-rgb1 krb5kdc[19927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:38 dist-test-slave-rgb1 krb5kdc[19927](info): set up 2 sockets
Aug 09 22:42:38 dist-test-slave-rgb1 krb5kdc[19927](info): commencing operation
krb5kdc: starting...
W20260809 22:42:39.233816 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.001s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:39 dist-test-slave-rgb1 krb5kdc[19927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:39.605167 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:39.966297 19941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:39.966681 19941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:39.966846 19941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:39.974722 19941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:39.974932 19941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:39.975069 19941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:39.975194 19941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:39.975283 19941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:39.986231 19941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:39.987728 19941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:39.989253 19941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:40.003527 19947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:40.003413 19949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:40.003883 19946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:40.004712 19941 server_base.cc:1034] running on GCE node
I20260809 22:42:40.005702 19941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:40.007479 19941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:40.008766 19941 hybrid_clock.cc:648] HybridClock initialized: now 1786315360008697 us; error 61 us; skew 500 ppm
I20260809 22:42:40.009238 19941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:40 dist-test-slave-rgb1 krb5kdc[19927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:40.013018 19941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:40.014699 19941 webserver.cc:460] Webserver started at http://127.0.0.1:38115/ using document root <none> and password file <none>
I20260809 22:42:40.015398 19941 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:40.015515 19941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:40.015787 19941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:40.018996 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "f6d99313319542a09c975fbf92c22173"
format_stamp: "Formatted at 2026-08-09 22:42:40 on dist-test-slave-rgb1"
server_key: "d5ce8d35195c1a39d1d8c33b3ab982bd"
server_key_iv: "142faddfb052c053ef3e740905769590"
server_key_version: "encryptionkey@0"
I20260809 22:42:40.019785 19941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "f6d99313319542a09c975fbf92c22173"
format_stamp: "Formatted at 2026-08-09 22:42:40 on dist-test-slave-rgb1"
server_key: "d5ce8d35195c1a39d1d8c33b3ab982bd"
server_key_iv: "142faddfb052c053ef3e740905769590"
server_key_version: "encryptionkey@0"
I20260809 22:42:40.025700 19941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 22:42:40.030103 19958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:40.031665 19941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:42:40.031863 19941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "f6d99313319542a09c975fbf92c22173"
format_stamp: "Formatted at 2026-08-09 22:42:40 on dist-test-slave-rgb1"
server_key: "d5ce8d35195c1a39d1d8c33b3ab982bd"
server_key_iv: "142faddfb052c053ef3e740905769590"
server_key_version: "encryptionkey@0"
I20260809 22:42:40.032078 19941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:40.038650 19941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:40.043576 19941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:40.043967 19941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:40.068606 19941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41907
I20260809 22:42:40.068719 20011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41907 every 8 connection(s)
I20260809 22:42:40.070262 19941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:40.071503 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 19941
I20260809 22:42:40.071743 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:40.072240 29645 external_mini_cluster.cc:1428] Setting key ffe4a71f33763013fbf2e9111093a897
I20260809 22:42:40.075101 20012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:40.082616 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:40.074443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:54483 (local address 127.0.0.1:41907)
0809 22:42:40.075711 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:40.075728 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:40.075784 (+ 56us) server_negotiation.cc:407] Connection header received
0809 22:42:40.078216 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:40.078247 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:40.078751 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:40.079291 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:40.080428 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:40.080436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:40.080452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:40.080484 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:40.080599 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:40.080630 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:40.080882 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:40.081037 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 22:42:40.081286 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":972,"thread_start_us":760,"threads_started":1}
I20260809 22:42:40.087399 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:40.089908 20012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173: Bootstrap starting.
I20260809 22:42:40.093717 20012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:40.095212 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:40.092324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:55343 (local address 127.0.0.1:41907)
0809 22:42:40.092523 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:40.092529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:40.092909 (+ 380us) server_negotiation.cc:407] Connection header received
0809 22:42:40.093065 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:40.093070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:40.093136 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:40.093249 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:40.094410 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:40.094416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:40.094420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:40.094440 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:40.094487 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:40.094506 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:40.094637 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:40.094813 (+ 176us) server_negotiation.cc:299] Negotiation successful
0809 22:42:40.094881 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 22:42:40.095180 20012 log.cc:824] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:40.098501 20012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173: No bootstrap required, opened a new log
I20260809 22:42:40.106446 20012 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d99313319542a09c975fbf92c22173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260809 22:42:40.106844 20012 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:40.106921 20012 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d99313319542a09c975fbf92c22173, State: Initialized, Role: FOLLOWER
I20260809 22:42:40.107518 20012 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d99313319542a09c975fbf92c22173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260809 22:42:40.107743 20012 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:40.107853 20012 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:40.107980 20012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:40.110437 20012 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d99313319542a09c975fbf92c22173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260809 22:42:40.111022 20012 leader_election.cc:302] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d99313319542a09c975fbf92c22173; no voters:
I20260809 22:42:40.111847 20012 leader_election.cc:288] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:40.112282 20022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:40.113584 20022 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [term 1 LEADER]: Becoming Leader. State: Replica: f6d99313319542a09c975fbf92c22173, State: Running, Role: LEADER
I20260809 22:42:40.114348 20022 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d99313319542a09c975fbf92c22173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } }
I20260809 22:42:40.114826 20012 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:40.117708 20023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6d99313319542a09c975fbf92c22173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d99313319542a09c975fbf92c22173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } } }
I20260809 22:42:40.117699 20024 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6d99313319542a09c975fbf92c22173. Latest consensus state: current_term: 1 leader_uuid: "f6d99313319542a09c975fbf92c22173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d99313319542a09c975fbf92c22173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41907 } } }
I20260809 22:42:40.118235 20023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:40.118247 20024 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:40.121886 20028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:40.126971 20028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:40.137602 20028 catalog_manager.cc:1347] Generated new cluster ID: 1c17edfc08064503ade6fb0f66933a6e
I20260809 22:42:40.137744 20028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:40.151302 20028 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:40.151782 20028 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:40.166414 20028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6d99313319542a09c975fbf92c22173: Generated new TSK 0
I20260809 22:42:40.167356 20028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:40.172865 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19941
W20260809 22:42:40.174237 20007 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.19:54483
W20260809 22:42:40.175060 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 19941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:40.175239 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 19941
2026-08-09T22:42:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2149 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:40 dist-test-slave-rgb1 krb5kdc[20043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:40 dist-test-slave-rgb1 krb5kdc[20043](info): set up 2 sockets
Aug 09 22:42:40 dist-test-slave-rgb1 krb5kdc[20043](info): commencing operation
krb5kdc: starting...
W20260809 22:42:41.439126 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.003s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:41 dist-test-slave-rgb1 krb5kdc[20043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:41.814098 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:42.185308 20057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:42.185734 20057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:42.185935 20057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:42.193640 20057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:42.193946 20057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:42.194087 20057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:42.194216 20057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:42.194324 20057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:42.205679 20057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:42.207245 20057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:42.208706 20057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:42.222777 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:42.222820 20063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:42.222791 20064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:42.223991 20057 server_base.cc:1034] running on GCE node
I20260809 22:42:42.224997 20057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:42.226619 20057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:42.227885 20057 hybrid_clock.cc:648] HybridClock initialized: now 1786315362227792 us; error 60 us; skew 500 ppm
I20260809 22:42:42.228334 20057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:42 dist-test-slave-rgb1 krb5kdc[20043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:42.235598 20057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:42.237329 20057 webserver.cc:460] Webserver started at http://127.0.0.1:45907/ using document root <none> and password file <none>
I20260809 22:42:42.238013 20057 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:42.238121 20057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:42.238374 20057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:42.241597 20057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "1e08cac9e4464c7d8a6695681dc02401"
format_stamp: "Formatted at 2026-08-09 22:42:42 on dist-test-slave-rgb1"
server_key: "22f6b00fa845ac7d450aec86e116d67e"
server_key_iv: "71d6afb2806b83f7602e821df9becfca"
server_key_version: "encryptionkey@0"
I20260809 22:42:42.242365 20057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "1e08cac9e4464c7d8a6695681dc02401"
format_stamp: "Formatted at 2026-08-09 22:42:42 on dist-test-slave-rgb1"
server_key: "22f6b00fa845ac7d450aec86e116d67e"
server_key_iv: "71d6afb2806b83f7602e821df9becfca"
server_key_version: "encryptionkey@0"
I20260809 22:42:42.247850 20057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 22:42:42.251694 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:42.253398 20057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:42:42.253618 20057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "1e08cac9e4464c7d8a6695681dc02401"
format_stamp: "Formatted at 2026-08-09 22:42:42 on dist-test-slave-rgb1"
server_key: "22f6b00fa845ac7d450aec86e116d67e"
server_key_iv: "71d6afb2806b83f7602e821df9becfca"
server_key_version: "encryptionkey@0"
I20260809 22:42:42.253862 20057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:42.292150 20057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:42.297629 20057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:42.297999 20057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:42.320550 20057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46777
I20260809 22:42:42.320597 20129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46777 every 8 connection(s)
I20260809 22:42:42.322160 20057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:42.326085 20130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:42.331918 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20057
I20260809 22:42:42.332257 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:42.332887 29645 external_mini_cluster.cc:1428] Setting key 08dc9a25826f86576f20c6accb3cfc54
I20260809 22:42:42.338546 20130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401: Bootstrap starting.
I20260809 22:42:42.342083 20130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:42.343247 20130 log.cc:824] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:42.346597 20130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401: No bootstrap required, opened a new log
Aug 09 22:42:42 dist-test-slave-rgb1 krb5kdc[20043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 22:42:42.357338 20130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e08cac9e4464c7d8a6695681dc02401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260809 22:42:42.357851 20130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:42.358143 20130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e08cac9e4464c7d8a6695681dc02401, State: Initialized, Role: FOLLOWER
I20260809 22:42:42.359118 20130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e08cac9e4464c7d8a6695681dc02401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260809 22:42:42.359432 20130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:42.359647 20130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:42.359858 20130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:42.360770 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:42.335009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:37645 (local address 127.0.0.1:46777)
0809 22:42:42.335897 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:42.335909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:42.335998 (+ 89us) server_negotiation.cc:407] Connection header received
0809 22:42:42.337831 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:42.337856 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:42.338309 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:42.338735 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:42.340528 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:42.342198 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:42.343875 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:42.344825 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:42.345043 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:42.349807 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:42.349819 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:42.349841 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:42.349889 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:42.354295 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:42.355780 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:42.355790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:42.355800 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:42.356053 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:42.356778 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:42.356785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:42.356790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:42.357550 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:42.357987 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:42.358723 (+ 736us) server_negotiation.cc:299] Negotiation successful
0809 22:42:42.359128 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":366,"threads_started":1}
I20260809 22:42:42.363417 20130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e08cac9e4464c7d8a6695681dc02401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260809 22:42:42.364214 20130 leader_election.cc:302] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e08cac9e4464c7d8a6695681dc02401; no voters:
I20260809 22:42:42.365077 20130 leader_election.cc:288] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:42.365198 20136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:42.365979 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:42.365927 20136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [term 1 LEADER]: Becoming Leader. State: Replica: 1e08cac9e4464c7d8a6695681dc02401, State: Running, Role: LEADER
I20260809 22:42:42.366515 20136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e08cac9e4464c7d8a6695681dc02401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } }
I20260809 22:42:42.368333 20136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e08cac9e4464c7d8a6695681dc02401. Latest consensus state: current_term: 1 leader_uuid: "1e08cac9e4464c7d8a6695681dc02401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e08cac9e4464c7d8a6695681dc02401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } } }
I20260809 22:42:42.368681 20136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:42.369316 20138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e08cac9e4464c7d8a6695681dc02401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e08cac9e4464c7d8a6695681dc02401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46777 } } }
I20260809 22:42:42.369724 20138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:42.379674 20130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:42.380110 20140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:42.389249 20140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:42.394546 20133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:42.379616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:48941 (local address 127.0.0.1:46777)
0809 22:42:42.379810 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:42.379818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:42.380294 (+ 476us) server_negotiation.cc:407] Connection header received
0809 22:42:42.380522 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:42.380528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:42.380641 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:42.380769 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:42.382360 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:42.383197 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:42.384512 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:42.385106 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:42.385267 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:42.387889 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:42.387894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:42.387897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:42.387931 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:42.391079 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:42.392028 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:42.392036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:42.392040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:42.392262 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:42.392899 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:42.392904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:42.392908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:42.393226 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:42.393515 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:42.393932 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 22:42:42.394136 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 22:42:42.413475 20140 catalog_manager.cc:1347] Generated new cluster ID: 5dd1832344f64e38a32c8736c2c9ecb7
I20260809 22:42:42.413630 20140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:42.447432 20140 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:42.449656 20140 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:42.462616 20140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e08cac9e4464c7d8a6695681dc02401: Generated new TSK 0
I20260809 22:42:42.463865 20140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:42.483708 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20057
I20260809 22:42:42.485476 20124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:42.583568 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20057
W20260809 22:42:42.605786 20041 connection.cc:535] client connection to 127.0.0.1:46777 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46777): Connection reset by peer (error 104)
2026-08-09T22:42:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:42 dist-test-slave-rgb1 krb5kdc[20163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:42 dist-test-slave-rgb1 krb5kdc[20163](info): set up 2 sockets
Aug 09 22:42:42 dist-test-slave-rgb1 krb5kdc[20163](info): commencing operation
krb5kdc: starting...
W20260809 22:42:43.806314 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:44 dist-test-slave-rgb1 krb5kdc[20163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:44.165753 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:44.525954 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:44.526291 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:44.526393 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:44.534207 20177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:44.534339 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:44.534389 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:44.534444 20177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:44.534482 20177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:44.544620 20177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:44.546012 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:44.547298 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:44.560281 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:44.560333 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:44.560257 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:44.561233 20177 server_base.cc:1034] running on GCE node
I20260809 22:42:44.562033 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:44.563567 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:44.564821 20177 hybrid_clock.cc:648] HybridClock initialized: now 1786315364564772 us; error 52 us; skew 500 ppm
I20260809 22:42:44.565295 20177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:44 dist-test-slave-rgb1 krb5kdc[20163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:44.569527 20177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:44.571849 20177 webserver.cc:460] Webserver started at http://127.0.0.1:42031/ using document root <none> and password file <none>
I20260809 22:42:44.572705 20177 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:44.572858 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:44.573343 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:44.576792 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85"
format_stamp: "Formatted at 2026-08-09 22:42:44 on dist-test-slave-rgb1"
server_key: "6ad8a2817bb17b7769c602bc9d6ab49f"
server_key_iv: "0189dad24b61a76fec4cede125e8cdc2"
server_key_version: "encryptionkey@0"
I20260809 22:42:44.577637 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85"
format_stamp: "Formatted at 2026-08-09 22:42:44 on dist-test-slave-rgb1"
server_key: "6ad8a2817bb17b7769c602bc9d6ab49f"
server_key_iv: "0189dad24b61a76fec4cede125e8cdc2"
server_key_version: "encryptionkey@0"
I20260809 22:42:44.583508 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:42:44.587592 20193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:44.589370 20177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:42:44.589603 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85"
format_stamp: "Formatted at 2026-08-09 22:42:44 on dist-test-slave-rgb1"
server_key: "6ad8a2817bb17b7769c602bc9d6ab49f"
server_key_iv: "0189dad24b61a76fec4cede125e8cdc2"
server_key_version: "encryptionkey@0"
I20260809 22:42:44.589848 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:44.624249 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:44.629153 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:44.629489 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:44.650063 20177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46485
I20260809 22:42:44.650123 20248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46485 every 8 connection(s)
I20260809 22:42:44.651641 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:44.652253 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20177
I20260809 22:42:44.652486 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:44.653220 29645 external_mini_cluster.cc:1428] Setting key 40f288ab519b515d43ec2896b7409eb5
I20260809 22:42:44.656224 20249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:44.667559 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:44.655233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:55951 (local address 127.0.0.1:46485)
0809 22:42:44.656039 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:44.656051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:44.656095 (+ 44us) server_negotiation.cc:407] Connection header received
0809 22:42:44.657833 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:44.657854 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:44.658237 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:44.658553 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:44.659225 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:44.660681 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:44.662619 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:44.663373 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:44.663534 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:44.664990 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:44.664999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:44.665014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:44.665593 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:44.665674 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:44.665698 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:44.665839 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:44.666246 (+ 407us) server_negotiation.cc:299] Negotiation successful
0809 22:42:44.666449 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":356,"threads_started":1}
I20260809 22:42:44.668956 20249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85: Bootstrap starting.
I20260809 22:42:44.671658 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:44.672325 20249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:44.673589 20249 log.cc:824] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:44.677336 20249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85: No bootstrap required, opened a new log
I20260809 22:42:44.682898 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:44.676441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:46509 (local address 127.0.0.1:46485)
0809 22:42:44.676684 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:44.676693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:44.677096 (+ 403us) server_negotiation.cc:407] Connection header received
0809 22:42:44.677331 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:44.677335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:44.677398 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:44.677509 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:44.678068 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:44.678852 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:44.680231 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:44.680777 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:44.680928 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:44.682115 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:44.682120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:44.682123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:44.682140 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:44.682181 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:44.682196 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:44.682327 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:44.682543 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 22:42:44.682607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 22:42:44.687063 20249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260809 22:42:44.687582 20249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:44.687709 20249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ecdd4fc8ef4a4a8974ef0ad4a4ff85, State: Initialized, Role: FOLLOWER
I20260809 22:42:44.688390 20249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260809 22:42:44.688721 20249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:44.688918 20249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:44.689122 20249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:44.692236 20249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260809 22:42:44.692930 20249 leader_election.cc:302] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15ecdd4fc8ef4a4a8974ef0ad4a4ff85; no voters:
I20260809 22:42:44.693742 20249 leader_election.cc:288] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:44.693893 20259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:44.695228 20259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [term 1 LEADER]: Becoming Leader. State: Replica: 15ecdd4fc8ef4a4a8974ef0ad4a4ff85, State: Running, Role: LEADER
I20260809 22:42:44.695766 20259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260809 22:42:44.696298 20249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:44.699062 20260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } } }
I20260809 22:42:44.699010 20261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15ecdd4fc8ef4a4a8974ef0ad4a4ff85. Latest consensus state: current_term: 1 leader_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ecdd4fc8ef4a4a8974ef0ad4a4ff85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } } }
I20260809 22:42:44.699483 20260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:44.699483 20261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:44.700433 20269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:44.708855 20269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:44.721745 20269 catalog_manager.cc:1347] Generated new cluster ID: e25b60ee554d4233bbdb61096b150b00
I20260809 22:42:44.721922 20269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:44.736335 20269 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:44.738533 20269 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:44.745103 20269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15ecdd4fc8ef4a4a8974ef0ad4a4ff85: Generated new TSK 0
I20260809 22:42:44.746096 20269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:44.752925 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20177
W20260809 22:42:44.754886 20244 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.19:55951
W20260809 22:42:44.755755 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:44.755930 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20177
W20260809 22:42:44.777997 20161 connection.cc:535] client connection to 127.0.0.1:46485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46485): Connection reset by peer (error 104)
2026-08-09T22:42:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2167 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:44 dist-test-slave-rgb1 krb5kdc[20281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:44 dist-test-slave-rgb1 krb5kdc[20281](info): set up 2 sockets
Aug 09 22:42:44 dist-test-slave-rgb1 krb5kdc[20281](info): commencing operation
krb5kdc: starting...
W20260809 22:42:46.014524 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:46 dist-test-slave-rgb1 krb5kdc[20281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 22:42:46.385015 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:46.782508 20294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:46.782897 20294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:46.783098 20294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:46.790621 20294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:46.790802 20294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:46.790917 20294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:46.791035 20294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:46.791126 20294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:46.801615 20294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:46.803135 20294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:46.804540 20294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:46.817708 20303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:46.817691 20300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:46.818178 20299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:46.818836 20294 server_base.cc:1034] running on GCE node
I20260809 22:42:46.819885 20294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:46.821691 20294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:46.823088 20294 hybrid_clock.cc:648] HybridClock initialized: now 1786315366823023 us; error 58 us; skew 500 ppm
I20260809 22:42:46.823621 20294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:46 dist-test-slave-rgb1 krb5kdc[20281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:46.827390 20294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 22:42:46.829185 20294 webserver.cc:460] Webserver started at http://127.0.0.1:38727/ using document root <none> and password file <none>
I20260809 22:42:46.829919 20294 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:46.830034 20294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:46.830297 20294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:46.833720 20294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "a5a35833857242ecbcf3ca8a3cd28ef0"
format_stamp: "Formatted at 2026-08-09 22:42:46 on dist-test-slave-rgb1"
server_key: "be263565a37c5b7b718dbecb1521a6e0"
server_key_iv: "38d5527ddf729aee72f948229caba333"
server_key_version: "encryptionkey@0"
I20260809 22:42:46.834823 20294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "a5a35833857242ecbcf3ca8a3cd28ef0"
format_stamp: "Formatted at 2026-08-09 22:42:46 on dist-test-slave-rgb1"
server_key: "be263565a37c5b7b718dbecb1521a6e0"
server_key_iv: "38d5527ddf729aee72f948229caba333"
server_key_version: "encryptionkey@0"
I20260809 22:42:46.841012 20294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260809 22:42:46.845391 20310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:46.847146 20294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 22:42:46.847321 20294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "a5a35833857242ecbcf3ca8a3cd28ef0"
format_stamp: "Formatted at 2026-08-09 22:42:46 on dist-test-slave-rgb1"
server_key: "be263565a37c5b7b718dbecb1521a6e0"
server_key_iv: "38d5527ddf729aee72f948229caba333"
server_key_version: "encryptionkey@0"
I20260809 22:42:46.847517 20294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:46.854239 20294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:46.859282 20294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:46.859584 20294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:46.881562 20294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32969
I20260809 22:42:46.881695 20363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32969 every 8 connection(s)
I20260809 22:42:46.883118 20294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:46.887427 20364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:46.892984 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20294
I20260809 22:42:46.893231 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:46.893772 29645 external_mini_cluster.cc:1428] Setting key 940c1f4f895671515ba794e13f0b8cca
I20260809 22:42:46.899431 20364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0: Bootstrap starting.
I20260809 22:42:46.903002 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:46.896136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:51689 (local address 127.0.0.1:32969)
0809 22:42:46.897175 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:46.897188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:46.897238 (+ 50us) server_negotiation.cc:407] Connection header received
0809 22:42:46.899123 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:46.899150 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:46.899689 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:46.900116 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:46.901264 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:46.901274 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:46.901287 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:46.901302 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:46.901367 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:46.901398 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:46.901576 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:46.901710 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 22:42:46.901931 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":515,"threads_started":1}
I20260809 22:42:46.903613 20364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:46.904971 20364 log.cc:824] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:46.907886 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:46.908480 20364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0: No bootstrap required, opened a new log
I20260809 22:42:46.917549 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:46.914346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:52365 (local address 127.0.0.1:32969)
0809 22:42:46.914619 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:46.914630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:46.915082 (+ 452us) server_negotiation.cc:407] Connection header received
0809 22:42:46.915218 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:46.915226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:46.915290 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:46.915403 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:46.916547 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:46.916551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:46.916621 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:46.916647 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:46.916714 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:46.916735 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:46.916861 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:46.917106 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 22:42:46.917226 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260809 22:42:46.919519 20364 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260809 22:42:46.919898 20364 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:46.920022 20364 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a35833857242ecbcf3ca8a3cd28ef0, State: Initialized, Role: FOLLOWER
I20260809 22:42:46.920702 20364 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260809 22:42:46.920969 20364 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:46.921123 20364 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:46.921320 20364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:46.923787 20364 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260809 22:42:46.924367 20364 leader_election.cc:302] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5a35833857242ecbcf3ca8a3cd28ef0; no voters:
I20260809 22:42:46.925060 20364 leader_election.cc:288] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:46.925226 20375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:46.926689 20375 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [term 1 LEADER]: Becoming Leader. State: Replica: a5a35833857242ecbcf3ca8a3cd28ef0, State: Running, Role: LEADER
I20260809 22:42:46.927213 20375 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260809 22:42:46.927465 20364 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:46.930192 20377 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a35833857242ecbcf3ca8a3cd28ef0. Latest consensus state: current_term: 1 leader_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260809 22:42:46.930686 20377 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:46.930621 20376 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a35833857242ecbcf3ca8a3cd28ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260809 22:42:46.930924 20376 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:46.936544 20381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:46.943082 20381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:46.954268 20381 catalog_manager.cc:1347] Generated new cluster ID: f333a43d842c409da28479cc030fb520
I20260809 22:42:46.954424 20381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:46.967640 20381 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:46.968184 20381 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:46.978958 20381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5a35833857242ecbcf3ca8a3cd28ef0: Generated new TSK 0
I20260809 22:42:46.980149 20381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:46.988260 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20294
W20260809 22:42:46.989619 20359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.19:51689
W20260809 22:42:46.990535 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20294: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:46.990715 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20294
2026-08-09T22:42:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:47 dist-test-slave-rgb1 krb5kdc[20399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:47 dist-test-slave-rgb1 krb5kdc[20399](info): set up 2 sockets
Aug 09 22:42:47 dist-test-slave-rgb1 krb5kdc[20399](info): commencing operation
krb5kdc: starting...
W20260809 22:42:48.206408 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:48 dist-test-slave-rgb1 krb5kdc[20399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:48.580850 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33303
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:35207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:48.941334 20412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:48.941674 20412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:48.941910 20412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:48.950249 20412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:48.950383 20412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:48.950502 20412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:48.950611 20412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:48.950676 20412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:48.961634 20412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:35207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:33303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:48.963038 20412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:48.964464 20412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:48.977043 20419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:48.977108 20418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:48.977147 20421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:48.978263 20412 server_base.cc:1034] running on GCE node
I20260809 22:42:48.979147 20412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:48.980772 20412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:48.982064 20412 hybrid_clock.cc:648] HybridClock initialized: now 1786315368982016 us; error 39 us; skew 500 ppm
I20260809 22:42:48.982528 20412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:48 dist-test-slave-rgb1 krb5kdc[20399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315368, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:48.989985 20412 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:42:48.991819 20412 webserver.cc:460] Webserver started at http://127.28.243.126:45863/ using document root <none> and password file <none>
I20260809 22:42:48.992650 20412 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:48.992781 20412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:48.993096 20412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:48.996449 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "fd69d3cad04b4852b38310566cb7914b"
format_stamp: "Formatted at 2026-08-09 22:42:48 on dist-test-slave-rgb1"
server_key: "6eb618e353517215cc22c6fdb6d4798b"
server_key_iv: "cdc5955f57e71aa778d1ebc01cda2ed6"
server_key_version: "encryptionkey@0"
I20260809 22:42:48.997380 20412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "fd69d3cad04b4852b38310566cb7914b"
format_stamp: "Formatted at 2026-08-09 22:42:48 on dist-test-slave-rgb1"
server_key: "6eb618e353517215cc22c6fdb6d4798b"
server_key_iv: "cdc5955f57e71aa778d1ebc01cda2ed6"
server_key_version: "encryptionkey@0"
I20260809 22:42:49.004083 20412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260809 22:42:49.008792 20428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:49.010403 20412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:42:49.010638 20412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "fd69d3cad04b4852b38310566cb7914b"
format_stamp: "Formatted at 2026-08-09 22:42:48 on dist-test-slave-rgb1"
server_key: "6eb618e353517215cc22c6fdb6d4798b"
server_key_iv: "cdc5955f57e71aa778d1ebc01cda2ed6"
server_key_version: "encryptionkey@0"
I20260809 22:42:49.010877 20412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:49.033641 20412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:49.038758 20412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:49.039108 20412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:49.060811 20412 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:33303
I20260809 22:42:49.060832 20481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:33303 every 8 connection(s)
I20260809 22:42:49.062438 20412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:49.066406 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:49.068329 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20412
I20260809 22:42:49.068596 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:49.069208 29645 external_mini_cluster.cc:1428] Setting key 449c32c9797b583fe608ecd79cfe53a1
I20260809 22:42:49.081861 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b: Bootstrap starting.
Aug 09 22:42:49 dist-test-slave-rgb1 krb5kdc[20399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:49.086225 20482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:49.087623 20482 log.cc:824] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:49.091038 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b: No bootstrap required, opened a new log
I20260809 22:42:49.094676 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:49.071203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56557 (local address 127.28.243.126:33303)
0809 22:42:49.072136 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:49.072147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:49.072192 (+ 45us) server_negotiation.cc:407] Connection header received
0809 22:42:49.073904 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:49.073928 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:49.074359 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:49.074717 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:49.076643 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:49.077986 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:49.079812 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:49.080475 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:49.080682 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:49.086377 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:49.086389 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:49.086411 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:49.086465 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:49.089959 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:49.090909 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:49.090916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:49.090922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:49.091112 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:49.091779 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:49.091783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:49.091785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:49.092335 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:49.092689 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:49.093177 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 22:42:49.093571 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":477,"threads_started":1}
I20260809 22:42:49.099071 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:49.100980 20482 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd69d3cad04b4852b38310566cb7914b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33303 } }
I20260809 22:42:49.101442 20482 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:49.101583 20482 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd69d3cad04b4852b38310566cb7914b, State: Initialized, Role: FOLLOWER
I20260809 22:42:49.102268 20482 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd69d3cad04b4852b38310566cb7914b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33303 } }
I20260809 22:42:49.102628 20482 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:49.102813 20482 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:49.103005 20482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:49.106124 20482 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd69d3cad04b4852b38310566cb7914b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33303 } }
I20260809 22:42:49.106657 20482 leader_election.cc:302] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd69d3cad04b4852b38310566cb7914b; no voters:
I20260809 22:42:49.107275 20482 leader_election.cc:288] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:49.107410 20493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:49.108897 20493 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [term 1 LEADER]: Becoming Leader. State: Replica: fd69d3cad04b4852b38310566cb7914b, State: Running, Role: LEADER
I20260809 22:42:49.109485 20493 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd69d3cad04b4852b38310566cb7914b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33303 } }
I20260809 22:42:49.109979 20482 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:49.113149 20495 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd69d3cad04b4852b38310566cb7914b. Latest consensus state: current_term: 1 leader_uuid: "fd69d3cad04b4852b38310566cb7914b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd69d3cad04b4852b38310566cb7914b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33303 } } }
I20260809 22:42:49.113603 20495 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:49.114208 20494 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd69d3cad04b4852b38310566cb7914b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd69d3cad04b4852b38310566cb7914b" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 33303 } } }
I20260809 22:42:49.114504 20494 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:49.117439 20499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:49.124809 20499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:49.125528 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:49.104854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42317 (local address 127.28.243.126:33303)
0809 22:42:49.105061 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:49.105068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:49.105545 (+ 477us) server_negotiation.cc:407] Connection header received
0809 22:42:49.105692 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:49.105698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:49.105776 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:49.105887 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:49.107799 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:49.109191 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:49.111134 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:49.112315 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:49.112483 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:49.116766 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:49.116777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:49.116785 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:49.116851 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:49.120492 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:49.121793 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:49.121798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:49.121802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:49.122007 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:49.122675 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:49.122679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:49.122683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:49.123158 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:49.124720 (+ 1562us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:49.124914 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 22:42:49.125134 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 22:42:49.140162 20499 catalog_manager.cc:1347] Generated new cluster ID: 8dc79ddb61ca408caa8b9ac862f28af4
I20260809 22:42:49.140507 20499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:49.150125 20499 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:49.151628 20499 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:49.161135 20499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd69d3cad04b4852b38310566cb7914b: Generated new TSK 0
I20260809 22:42:49.162253 20499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:49.170754 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20412
I20260809 22:42:49.172416 20477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:49.268738 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20412
W20260809 22:42:49.288951 20396 connection.cc:535] client connection to 127.28.243.126:33303 recv error: Network error: failed to read from TLS socket (remote: 127.28.243.126:33303): Connection reset by peer (error 104)
2026-08-09T22:42:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:49 dist-test-slave-rgb1 krb5kdc[20516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:49 dist-test-slave-rgb1 krb5kdc[20516](info): set up 2 sockets
Aug 09 22:42:49 dist-test-slave-rgb1 krb5kdc[20516](info): commencing operation
krb5kdc: starting...
W20260809 22:42:50.479025 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.005s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:50 dist-test-slave-rgb1 krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:50.849642 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36057
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:45927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:36057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:51.214248 20529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:51.214630 20529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:51.214805 20529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:51.222182 20529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:51.222345 20529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:51.222455 20529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:51.222587 20529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:51.222671 20529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:51.233078 20529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:45927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:36057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:36057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:51.234543 20529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:51.236019 20529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:51.249028 20540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:51.249073 20536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:51.248951 20538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:51.250237 20529 server_base.cc:1034] running on GCE node
I20260809 22:42:51.251004 20529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:51.252471 20529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:51.253732 20529 hybrid_clock.cc:648] HybridClock initialized: now 1786315371253671 us; error 54 us; skew 500 ppm
I20260809 22:42:51.254186 20529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:51 dist-test-slave-rgb1 krb5kdc[20516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315371, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:51.257946 20529 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:42:51.259784 20529 webserver.cc:460] Webserver started at http://127.28.243.126:34319/ using document root <none> and password file <none>
I20260809 22:42:51.260528 20529 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:51.260733 20529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:51.261080 20529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:51.264232 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "baa66a7517194e20b6a8240c08e59788"
format_stamp: "Formatted at 2026-08-09 22:42:51 on dist-test-slave-rgb1"
server_key: "476a102d5221ec831e108b96ed2e4d9e"
server_key_iv: "98f9205204e577d1a2d069296b0796da"
server_key_version: "encryptionkey@0"
I20260809 22:42:51.265161 20529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "baa66a7517194e20b6a8240c08e59788"
format_stamp: "Formatted at 2026-08-09 22:42:51 on dist-test-slave-rgb1"
server_key: "476a102d5221ec831e108b96ed2e4d9e"
server_key_iv: "98f9205204e577d1a2d069296b0796da"
server_key_version: "encryptionkey@0"
I20260809 22:42:51.270612 20529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:42:51.274534 20547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:51.276021 20529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 22:42:51.276356 20529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "baa66a7517194e20b6a8240c08e59788"
format_stamp: "Formatted at 2026-08-09 22:42:51 on dist-test-slave-rgb1"
server_key: "476a102d5221ec831e108b96ed2e4d9e"
server_key_iv: "98f9205204e577d1a2d069296b0796da"
server_key_version: "encryptionkey@0"
I20260809 22:42:51.276607 20529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:51.304311 20529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:51.309172 20529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:51.309502 20529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:51.332836 20529 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:36057
I20260809 22:42:51.332909 20598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:36057 every 8 connection(s)
I20260809 22:42:51.334575 20529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:51.336198 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20529
I20260809 22:42:51.336395 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:51.337001 29645 external_mini_cluster.cc:1428] Setting key 6d403a07780bc6a9343aa1bcc70467b4
I20260809 22:42:51.339619 20599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:51.351155 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:51.338854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44469 (local address 127.28.243.126:36057)
0809 22:42:51.339733 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:51.339748 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:51.339799 (+ 51us) server_negotiation.cc:407] Connection header received
0809 22:42:51.341525 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:51.341547 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:51.341918 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:51.342259 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:51.342928 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:51.344390 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:51.346322 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:51.347051 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:51.347192 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:51.348550 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:51.348626 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:51.348641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:51.349235 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:51.349311 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:51.349345 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:51.349480 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:51.349894 (+ 414us) server_negotiation.cc:299] Negotiation successful
0809 22:42:51.350161 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":372,"threads_started":1}
I20260809 22:42:51.352144 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788: Bootstrap starting.
I20260809 22:42:51.355597 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:51.356352 20599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:51.357725 20599 log.cc:824] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:51.361557 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788: No bootstrap required, opened a new log
I20260809 22:42:51.369318 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:51.361295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44279 (local address 127.28.243.126:36057)
0809 22:42:51.361495 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:51.361514 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:51.361836 (+ 322us) server_negotiation.cc:407] Connection header received
0809 22:42:51.362035 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:51.362041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:51.362141 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:51.362302 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:51.362819 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:51.364138 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:51.366051 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:51.366809 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:51.366965 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:51.368312 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:51.368318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:51.368323 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:51.368353 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:51.368404 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:51.368429 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:51.368643 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:51.368868 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 22:42:51.368927 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:42:51.372298 20599 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa66a7517194e20b6a8240c08e59788" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36057 } }
I20260809 22:42:51.372817 20599 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:51.372932 20599 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa66a7517194e20b6a8240c08e59788, State: Initialized, Role: FOLLOWER
I20260809 22:42:51.373678 20599 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa66a7517194e20b6a8240c08e59788" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36057 } }
I20260809 22:42:51.373936 20599 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:51.374130 20599 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:51.374332 20599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:51.377247 20599 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa66a7517194e20b6a8240c08e59788" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36057 } }
I20260809 22:42:51.377832 20599 leader_election.cc:302] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa66a7517194e20b6a8240c08e59788; no voters:
I20260809 22:42:51.378515 20599 leader_election.cc:288] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:51.378702 20609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:51.380930 20599 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:51.384848 20609 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [term 1 LEADER]: Becoming Leader. State: Replica: baa66a7517194e20b6a8240c08e59788, State: Running, Role: LEADER
I20260809 22:42:51.385733 20609 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa66a7517194e20b6a8240c08e59788" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36057 } }
I20260809 22:42:51.388963 20610 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa66a7517194e20b6a8240c08e59788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa66a7517194e20b6a8240c08e59788" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36057 } } }
I20260809 22:42:51.389480 20610 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:51.389082 20611 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa66a7517194e20b6a8240c08e59788. Latest consensus state: current_term: 1 leader_uuid: "baa66a7517194e20b6a8240c08e59788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa66a7517194e20b6a8240c08e59788" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 36057 } } }
I20260809 22:42:51.390813 20611 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:51.402431 20623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:51.409466 20623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:51.419800 20623 catalog_manager.cc:1347] Generated new cluster ID: 97a42a9d01c44a4ab8426750519c457d
I20260809 22:42:51.419962 20623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:51.442073 20623 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:51.444169 20623 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:51.469435 20623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa66a7517194e20b6a8240c08e59788: Generated new TSK 0
I20260809 22:42:51.470404 20623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:51.488333 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20529
W20260809 22:42:51.489809 20594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44469
W20260809 22:42:51.490934 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20529: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:51.491113 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20529
W20260809 22:42:51.512310 20514 connection.cc:535] client connection to 127.28.243.126:36057 recv error: Network error: failed to read from TLS socket (remote: 127.28.243.126:36057): Connection reset by peer (error 104)
2026-08-09T22:42:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:51 dist-test-slave-rgb1 krb5kdc[20630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:51 dist-test-slave-rgb1 krb5kdc[20630](info): set up 2 sockets
Aug 09 22:42:51 dist-test-slave-rgb1 krb5kdc[20630](info): commencing operation
krb5kdc: starting...
W20260809 22:42:52.748708 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:52 dist-test-slave-rgb1 krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:53.109779 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35163
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:38703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:35163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:53.490922 20645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:53.491266 20645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:53.491430 20645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:53.500385 20645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:53.500547 20645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:53.500691 20645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:53.500805 20645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:53.500874 20645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:53.511698 20645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:38703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:35163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:53.513216 20645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:53.515034 20645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:53.530333 20650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:53.530296 20651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:53.530274 20653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:53.531338 20645 server_base.cc:1034] running on GCE node
I20260809 22:42:53.532469 20645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:53.534302 20645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:53.535585 20645 hybrid_clock.cc:648] HybridClock initialized: now 1786315373535534 us; error 41 us; skew 500 ppm
I20260809 22:42:53.536141 20645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:53 dist-test-slave-rgb1 krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315373, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:53.540138 20645 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:42:53.542147 20645 webserver.cc:460] Webserver started at http://127.28.243.126:40297/ using document root <none> and password file <none>
I20260809 22:42:53.542906 20645 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:53.543030 20645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:53.543321 20645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:53.546797 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "6401f486772c4b6385363113f7902ac1"
format_stamp: "Formatted at 2026-08-09 22:42:53 on dist-test-slave-rgb1"
server_key: "fc856b867c1bbbcb6b1e696ac80d63e1"
server_key_iv: "563d76972326a1edbee091fa1ea73235"
server_key_version: "encryptionkey@0"
I20260809 22:42:53.547638 20645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "6401f486772c4b6385363113f7902ac1"
format_stamp: "Formatted at 2026-08-09 22:42:53 on dist-test-slave-rgb1"
server_key: "fc856b867c1bbbcb6b1e696ac80d63e1"
server_key_iv: "563d76972326a1edbee091fa1ea73235"
server_key_version: "encryptionkey@0"
I20260809 22:42:53.554034 20645 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 22:42:53.558264 20660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:53.559926 20645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260809 22:42:53.560154 20645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "6401f486772c4b6385363113f7902ac1"
format_stamp: "Formatted at 2026-08-09 22:42:53 on dist-test-slave-rgb1"
server_key: "fc856b867c1bbbcb6b1e696ac80d63e1"
server_key_iv: "563d76972326a1edbee091fa1ea73235"
server_key_version: "encryptionkey@0"
I20260809 22:42:53.560393 20645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:53.566795 20645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:53.571421 20645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:53.571764 20645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:53.592809 20645 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:35163
I20260809 22:42:53.592811 20715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:35163 every 8 connection(s)
I20260809 22:42:53.594354 20645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:53.596674 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20645
I20260809 22:42:53.596879 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:53.597414 29645 external_mini_cluster.cc:1428] Setting key d6af41ac563191e141344340e22749cb
I20260809 22:42:53.598788 20716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:53.605916 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:53.599310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38351 (local address 127.28.243.126:35163)
0809 22:42:53.600166 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:53.600177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:53.600221 (+ 44us) server_negotiation.cc:407] Connection header received
0809 22:42:53.601991 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:53.602016 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:53.602401 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:53.602863 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:53.604119 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:53.604125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:53.604138 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:53.604156 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:53.604230 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:53.604253 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:53.604532 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:53.604735 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 22:42:53.604940 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":395,"threads_started":1}
I20260809 22:42:53.610297 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:53.611968 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1: Bootstrap starting.
I20260809 22:42:53.615918 20716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:53.617282 20716 log.cc:824] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:53.618265 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:53.615332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59297 (local address 127.28.243.126:35163)
0809 22:42:53.615502 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:53.615506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:53.615967 (+ 461us) server_negotiation.cc:407] Connection header received
0809 22:42:53.616223 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:53.616227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:53.616280 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:53.616401 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:53.617527 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:53.617539 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:53.617544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:53.617560 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:53.617603 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:53.617620 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:53.617757 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:53.617895 (+ 138us) server_negotiation.cc:299] Negotiation successful
0809 22:42:53.617985 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 22:42:53.620679 20716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1: No bootstrap required, opened a new log
I20260809 22:42:53.629263 20716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401f486772c4b6385363113f7902ac1" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35163 } }
I20260809 22:42:53.629715 20716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:53.629823 20716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6401f486772c4b6385363113f7902ac1, State: Initialized, Role: FOLLOWER
I20260809 22:42:53.630478 20716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401f486772c4b6385363113f7902ac1" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35163 } }
I20260809 22:42:53.630722 20716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:53.630887 20716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:53.631089 20716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:53.634054 20716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401f486772c4b6385363113f7902ac1" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35163 } }
I20260809 22:42:53.634595 20716 leader_election.cc:302] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6401f486772c4b6385363113f7902ac1; no voters:
I20260809 22:42:53.635246 20716 leader_election.cc:288] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:53.635417 20727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:53.636168 20727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [term 1 LEADER]: Becoming Leader. State: Replica: 6401f486772c4b6385363113f7902ac1, State: Running, Role: LEADER
I20260809 22:42:53.636817 20727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401f486772c4b6385363113f7902ac1" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35163 } }
I20260809 22:42:53.637594 20716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:53.641805 20729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6401f486772c4b6385363113f7902ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401f486772c4b6385363113f7902ac1" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35163 } } }
I20260809 22:42:53.642326 20729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:53.642506 20730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6401f486772c4b6385363113f7902ac1. Latest consensus state: current_term: 1 leader_uuid: "6401f486772c4b6385363113f7902ac1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6401f486772c4b6385363113f7902ac1" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35163 } } }
I20260809 22:42:53.642803 20730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:53.643513 20736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:53.649106 20736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:53.659101 20736 catalog_manager.cc:1347] Generated new cluster ID: d4f9da0885414b749b0452d065c1e761
I20260809 22:42:53.659230 20736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:53.689148 20736 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:53.689621 20736 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:53.697939 20736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6401f486772c4b6385363113f7902ac1: Generated new TSK 0
I20260809 22:42:53.698812 20736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:53.712883 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20645
W20260809 22:42:53.714188 20711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38351
W20260809 22:42:53.715128 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20645: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:53.715312 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20645
2026-08-09T22:42:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:53 dist-test-slave-rgb1 krb5kdc[20752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:53 dist-test-slave-rgb1 krb5kdc[20752](info): set up 2 sockets
Aug 09 22:42:53 dist-test-slave-rgb1 krb5kdc[20752](info): commencing operation
krb5kdc: starting...
W20260809 22:42:54.978346 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.008s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:55 dist-test-slave-rgb1 krb5kdc[20752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:55.341816 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34211
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:55.715389 20767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:55.715727 20767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:55.715903 20767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:55.723649 20767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:55.723794 20767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:55.723898 20767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:55.723989 20767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:55.724069 20767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:55.734697 20767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:34211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:55.736131 20767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:55.737942 20767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:55.750772 20773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:55.750882 20772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:55.750869 20775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:55.752185 20767 server_base.cc:1034] running on GCE node
I20260809 22:42:55.753093 20767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:55.754547 20767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:55.755795 20767 hybrid_clock.cc:648] HybridClock initialized: now 1786315375755752 us; error 39 us; skew 500 ppm
I20260809 22:42:55.756191 20767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:55 dist-test-slave-rgb1 krb5kdc[20752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315375, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:55.763298 20767 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:42:55.764967 20767 webserver.cc:460] Webserver started at http://127.28.243.126:33427/ using document root <none> and password file <none>
I20260809 22:42:55.765654 20767 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:55.765762 20767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:55.766016 20767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:55.769223 20767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "7ce6581c2140449cbc8c00527891695c"
format_stamp: "Formatted at 2026-08-09 22:42:55 on dist-test-slave-rgb1"
server_key: "916ed820b5b0ad732d862fe060605fcf"
server_key_iv: "aa67b42dd3faef4bcd53e3da267493d6"
server_key_version: "encryptionkey@0"
I20260809 22:42:55.769958 20767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "7ce6581c2140449cbc8c00527891695c"
format_stamp: "Formatted at 2026-08-09 22:42:55 on dist-test-slave-rgb1"
server_key: "916ed820b5b0ad732d862fe060605fcf"
server_key_iv: "aa67b42dd3faef4bcd53e3da267493d6"
server_key_version: "encryptionkey@0"
I20260809 22:42:55.775288 20767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:42:55.779129 20782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:55.780694 20767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 22:42:55.780887 20767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "7ce6581c2140449cbc8c00527891695c"
format_stamp: "Formatted at 2026-08-09 22:42:55 on dist-test-slave-rgb1"
server_key: "916ed820b5b0ad732d862fe060605fcf"
server_key_iv: "aa67b42dd3faef4bcd53e3da267493d6"
server_key_version: "encryptionkey@0"
I20260809 22:42:55.781101 20767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:55.798667 20767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:55.803040 20767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:55.803354 20767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:55.823503 20767 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:34211
I20260809 22:42:55.823522 20841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:34211 every 8 connection(s)
I20260809 22:42:55.825009 20767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:55.827976 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20767
I20260809 22:42:55.828202 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:55.828832 29645 external_mini_cluster.cc:1428] Setting key bb44f20a9f9a875907ac05ca4a4a75e5
I20260809 22:42:55.829628 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:55.842844 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c: Bootstrap starting.
Aug 09 22:42:55 dist-test-slave-rgb1 krb5kdc[20752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:42:55.846295 20843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:55.847607 20843 log.cc:824] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:55.850910 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c: No bootstrap required, opened a new log
I20260809 22:42:55.854138 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:55.830931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54105 (local address 127.28.243.126:34211)
0809 22:42:55.831765 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:55.831777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:55.831819 (+ 42us) server_negotiation.cc:407] Connection header received
0809 22:42:55.834447 (+ 2628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:55.834474 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:55.834978 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:55.835469 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:55.837134 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:55.838566 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:55.839921 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:55.840730 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:55.840942 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:55.845437 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:55.845445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:55.845460 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:55.845507 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:55.849272 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:55.850097 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:55.850110 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:55.850116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:55.850295 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:55.850695 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:55.850700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:55.850703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:55.851413 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:55.851832 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:55.852394 (+ 562us) server_negotiation.cc:299] Negotiation successful
0809 22:42:55.852937 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":366,"threads_started":1}
I20260809 22:42:55.858842 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:55.859865 20843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce6581c2140449cbc8c00527891695c" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34211 } }
I20260809 22:42:55.860275 20843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:55.860389 20843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce6581c2140449cbc8c00527891695c, State: Initialized, Role: FOLLOWER
I20260809 22:42:55.861078 20843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce6581c2140449cbc8c00527891695c" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34211 } }
I20260809 22:42:55.861325 20843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:55.861508 20843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:55.861716 20843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:55.865681 20843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce6581c2140449cbc8c00527891695c" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34211 } }
I20260809 22:42:55.866264 20843 leader_election.cc:302] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ce6581c2140449cbc8c00527891695c; no voters:
I20260809 22:42:55.866983 20843 leader_election.cc:288] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:55.867108 20854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:55.868912 20854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [term 1 LEADER]: Becoming Leader. State: Replica: 7ce6581c2140449cbc8c00527891695c, State: Running, Role: LEADER
I20260809 22:42:55.869885 20843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:55.869792 20854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce6581c2140449cbc8c00527891695c" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34211 } }
I20260809 22:42:55.873679 20855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ce6581c2140449cbc8c00527891695c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce6581c2140449cbc8c00527891695c" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34211 } } }
I20260809 22:42:55.874310 20855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:55.882028 20861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:55.882783 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:55.864358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41541 (local address 127.28.243.126:34211)
0809 22:42:55.864610 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:55.864619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:55.864925 (+ 306us) server_negotiation.cc:407] Connection header received
0809 22:42:55.865145 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:55.865150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:55.865216 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:55.865322 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:55.867044 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:55.868238 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:55.869909 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:55.870475 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:55.870625 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:55.873822 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:55.873829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:55.873834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:42:55.873882 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:55.878947 (+ 5065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:55.880196 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:55.880202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:55.880205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:55.880433 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:42:55.881005 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:42:55.881009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:42:55.881012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:42:55.881524 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:55.881822 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:55.882180 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 22:42:55.882440 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:42:55.885560 20856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ce6581c2140449cbc8c00527891695c. Latest consensus state: current_term: 1 leader_uuid: "7ce6581c2140449cbc8c00527891695c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ce6581c2140449cbc8c00527891695c" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 34211 } } }
I20260809 22:42:55.886148 20856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:55.891542 20861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:55.903110 20861 catalog_manager.cc:1347] Generated new cluster ID: b9471ac30286456d9639ca3b62482510
I20260809 22:42:55.903256 20861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:55.916821 20861 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:55.918900 20861 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:55.932528 20861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ce6581c2140449cbc8c00527891695c: Generated new TSK 0
I20260809 22:42:55.933714 20861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:55.947472 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20767
I20260809 22:42:55.948737 20837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:42:56.043979 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20767
2026-08-09T22:42:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:56 dist-test-slave-rgb1 krb5kdc[20877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:56 dist-test-slave-rgb1 krb5kdc[20877](info): set up 2 sockets
Aug 09 22:42:56 dist-test-slave-rgb1 krb5kdc[20877](info): commencing operation
krb5kdc: starting...
W20260809 22:42:57.274767 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:57 dist-test-slave-rgb1 krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:57.648990 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35345
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:43041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:42:58.019757 20890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:42:58.020160 20890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:42:58.020341 20890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:42:58.027839 20890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:42:58.028059 20890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:42:58.028165 20890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:42:58.028275 20890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:42:58.028357 20890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:42:58.038921 20890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:43041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:35345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:42:58.040326 20890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:42:58.041705 20890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:42:58.054270 20895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:58.054170 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:42:58.054210 20896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:42:58.055380 20890 server_base.cc:1034] running on GCE node
I20260809 22:42:58.056186 20890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:42:58.057789 20890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:42:58.059005 20890 hybrid_clock.cc:648] HybridClock initialized: now 1786315378058952 us; error 45 us; skew 500 ppm
I20260809 22:42:58.059340 20890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:42:58 dist-test-slave-rgb1 krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315378, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:42:58.062700 20890 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:42:58.064431 20890 webserver.cc:460] Webserver started at http://127.28.243.126:41187/ using document root <none> and password file <none>
I20260809 22:42:58.065178 20890 fs_manager.cc:359] Metadata directory not provided
I20260809 22:42:58.065321 20890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:42:58.065651 20890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:42:58.068832 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "c15795a802dc445a9d6bbed8759e60f4"
format_stamp: "Formatted at 2026-08-09 22:42:58 on dist-test-slave-rgb1"
server_key: "cc0b9f1cbd894b3a53424287dd8e1fe0"
server_key_iv: "65a7b927f8381e17b9627e4a1ba2a56d"
server_key_version: "encryptionkey@0"
I20260809 22:42:58.069636 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "c15795a802dc445a9d6bbed8759e60f4"
format_stamp: "Formatted at 2026-08-09 22:42:58 on dist-test-slave-rgb1"
server_key: "cc0b9f1cbd894b3a53424287dd8e1fe0"
server_key_iv: "65a7b927f8381e17b9627e4a1ba2a56d"
server_key_version: "encryptionkey@0"
I20260809 22:42:58.075103 20890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 22:42:58.078971 20906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:42:58.080659 20890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:42:58.080873 20890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "c15795a802dc445a9d6bbed8759e60f4"
format_stamp: "Formatted at 2026-08-09 22:42:58 on dist-test-slave-rgb1"
server_key: "cc0b9f1cbd894b3a53424287dd8e1fe0"
server_key_iv: "65a7b927f8381e17b9627e4a1ba2a56d"
server_key_version: "encryptionkey@0"
I20260809 22:42:58.081095 20890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:42:58.116127 20890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:42:58.121711 20890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:42:58.122066 20890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:42:58.144356 20890 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:35345
I20260809 22:42:58.144403 20963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:35345 every 8 connection(s)
I20260809 22:42:58.145923 20890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:42:58.150112 20964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:42:58.155579 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 20890
I20260809 22:42:58.155787 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:42:58.156378 29645 external_mini_cluster.cc:1428] Setting key e621b53697a36110796868adf7a435ca
I20260809 22:42:58.162786 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4: Bootstrap starting.
I20260809 22:42:58.166604 20964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4: Neither blocks nor log segments found. Creating new log.
I20260809 22:42:58.168212 20964 log.cc:824] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4: Log is configured to *not* fsync() on all Append() calls
I20260809 22:42:58.171442 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:58.158443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59807 (local address 127.28.243.126:35345)
0809 22:42:58.159340 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:58.159352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:58.159396 (+ 44us) server_negotiation.cc:407] Connection header received
0809 22:42:58.161270 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:58.161297 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:58.161697 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:58.162029 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:58.162654 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:58.164081 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:58.166093 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:58.166864 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:58.167014 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:58.168677 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:58.168687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:58.168702 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:58.169539 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:58.169691 (+ 152us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:58.169729 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:58.169875 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:58.170155 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 22:42:58.170417 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":409,"threads_started":1}
I20260809 22:42:58.171931 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4: No bootstrap required, opened a new log
I20260809 22:42:58.175580 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:42:58.182595 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15795a802dc445a9d6bbed8759e60f4" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35345 } }
I20260809 22:42:58.183149 20964 raft_consensus.cc:374] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:42:58.183266 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15795a802dc445a9d6bbed8759e60f4, State: Initialized, Role: FOLLOWER
I20260809 22:42:58.183975 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15795a802dc445a9d6bbed8759e60f4" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35345 } }
I20260809 22:42:58.184237 20964 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:42:58.184401 20964 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:42:58.184631 20964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:42:58.187605 20964 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15795a802dc445a9d6bbed8759e60f4" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35345 } }
I20260809 22:42:58.188268 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:42:58.181178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58007 (local address 127.28.243.126:35345)
0809 22:42:58.181377 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 22:42:58.181385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:42:58.181870 (+ 485us) server_negotiation.cc:407] Connection header received
0809 22:42:58.182063 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:42:58.182068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:42:58.182156 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:42:58.182269 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:42:58.182738 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:58.183839 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:58.185535 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:42:58.186280 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:42:58.186412 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:42:58.187496 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:42:58.187502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:42:58.187508 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:42:58.187535 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:42:58.187585 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:42:58.187614 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:42:58.187719 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 22:42:58.187851 (+ 132us) server_negotiation.cc:299] Negotiation successful
0809 22:42:58.187948 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 22:42:58.188272 20964 leader_election.cc:302] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15795a802dc445a9d6bbed8759e60f4; no voters:
I20260809 22:42:58.189172 20964 leader_election.cc:288] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:42:58.189283 20974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:42:58.190392 20974 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [term 1 LEADER]: Becoming Leader. State: Replica: c15795a802dc445a9d6bbed8759e60f4, State: Running, Role: LEADER
I20260809 22:42:58.191017 20974 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15795a802dc445a9d6bbed8759e60f4" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35345 } }
I20260809 22:42:58.191694 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:42:58.194064 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c15795a802dc445a9d6bbed8759e60f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15795a802dc445a9d6bbed8759e60f4" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35345 } } }
I20260809 22:42:58.194540 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:58.194782 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15795a802dc445a9d6bbed8759e60f4. Latest consensus state: current_term: 1 leader_uuid: "c15795a802dc445a9d6bbed8759e60f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15795a802dc445a9d6bbed8759e60f4" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 35345 } } }
I20260809 22:42:58.195113 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4 [sys.catalog]: This master's current role is: LEADER
I20260809 22:42:58.196898 20979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:42:58.204363 20979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:42:58.216548 20979 catalog_manager.cc:1347] Generated new cluster ID: 8bc6a736aa0d4ced950ab5c6794571bc
I20260809 22:42:58.216718 20979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:42:58.234754 20979 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:42:58.237030 20979 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:42:58.246698 20979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15795a802dc445a9d6bbed8759e60f4: Generated new TSK 0
I20260809 22:42:58.247867 20979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:42:58.257228 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20890
W20260809 22:42:58.259578 20957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59807
W20260809 22:42:58.260430 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 20890: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:42:58.260623 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 20890
2026-08-09T22:42:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:42:58 dist-test-slave-rgb1 krb5kdc[20996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:42:58 dist-test-slave-rgb1 krb5kdc[20996](info): set up 2 sockets
Aug 09 22:42:58 dist-test-slave-rgb1 krb5kdc[20996](info): commencing operation
krb5kdc: starting...
W20260809 22:42:59.492816 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.008s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:42:59 dist-test-slave-rgb1 krb5kdc[20996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:42:59.864171 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40847
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:40847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:43:00.234685 21009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:43:00.235005 21009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:43:00.235112 21009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:43:00.242703 21009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:43:00.242847 21009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:43:00.242898 21009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:43:00.242971 21009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:43:00.243018 21009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:43:00.253863 21009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:40847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:40847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:43:00.255823 21009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:43:00.257524 21009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:43:00.271400 21014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:00.271461 21017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:00.271363 21015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:43:00.274767 21009 server_base.cc:1034] running on GCE node
I20260809 22:43:00.275971 21009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:43:00.278263 21009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:43:00.279561 21009 hybrid_clock.cc:648] HybridClock initialized: now 1786315380279517 us; error 43 us; skew 500 ppm
I20260809 22:43:00.279965 21009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:43:00 dist-test-slave-rgb1 krb5kdc[20996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315380, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:43:00.283792 21009 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:43:00.285696 21009 webserver.cc:460] Webserver started at http://127.28.243.126:35473/ using document root <none> and password file <none>
I20260809 22:43:00.286473 21009 fs_manager.cc:359] Metadata directory not provided
I20260809 22:43:00.286634 21009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:43:00.286960 21009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:43:00.290361 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "aa850007cf1a4023bc131bf8d0c92804"
format_stamp: "Formatted at 2026-08-09 22:43:00 on dist-test-slave-rgb1"
server_key: "57a6ee35453744a7a2c25e03357036ed"
server_key_iv: "2da89dcbe082d98bf3fc10f3087bda71"
server_key_version: "encryptionkey@0"
I20260809 22:43:00.291213 21009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "aa850007cf1a4023bc131bf8d0c92804"
format_stamp: "Formatted at 2026-08-09 22:43:00 on dist-test-slave-rgb1"
server_key: "57a6ee35453744a7a2c25e03357036ed"
server_key_iv: "2da89dcbe082d98bf3fc10f3087bda71"
server_key_version: "encryptionkey@0"
I20260809 22:43:00.297760 21009 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260809 22:43:00.301982 21025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:43:00.303912 21009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260809 22:43:00.304147 21009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "aa850007cf1a4023bc131bf8d0c92804"
format_stamp: "Formatted at 2026-08-09 22:43:00 on dist-test-slave-rgb1"
server_key: "57a6ee35453744a7a2c25e03357036ed"
server_key_iv: "2da89dcbe082d98bf3fc10f3087bda71"
server_key_version: "encryptionkey@0"
I20260809 22:43:00.304384 21009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:43:00.311441 21009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:43:00.316274 21009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:43:00.316695 21009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:43:00.339601 21009 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:40847
I20260809 22:43:00.339649 21080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:40847 every 8 connection(s)
I20260809 22:43:00.341161 21009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:43:00.345424 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:43:00.350410 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 21009
I20260809 22:43:00.350656 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:43:00.351186 29645 external_mini_cluster.cc:1428] Setting key 7d8cc41f6f1d6e8d88e874291f5a1cc7
I20260809 22:43:00.357875 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804: Bootstrap starting.
I20260809 22:43:00.361128 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:00.353376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49937 (local address 127.28.243.126:40847)
0809 22:43:00.354465 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:00.354480 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:00.354550 (+ 70us) server_negotiation.cc:407] Connection header received
0809 22:43:00.356981 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:00.357010 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:00.357474 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:00.357968 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:00.359103 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:00.359111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:00.359127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:43:00.359154 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:00.359229 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:43:00.359257 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:00.359471 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:00.359633 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 22:43:00.359857 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":538,"threads_started":1}
I20260809 22:43:00.362111 21081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804: Neither blocks nor log segments found. Creating new log.
I20260809 22:43:00.363305 21081 log.cc:824] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804: Log is configured to *not* fsync() on all Append() calls
I20260809 22:43:00.365434 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:43:00.366472 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804: No bootstrap required, opened a new log
I20260809 22:43:00.372601 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:00.369991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50625 (local address 127.28.243.126:40847)
0809 22:43:00.370199 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:00.370206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:00.370593 (+ 387us) server_negotiation.cc:407] Connection header received
0809 22:43:00.370728 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:00.370733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:00.370813 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:00.370933 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:00.371908 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:00.371915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:00.371921 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 22:43:00.371945 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:00.371993 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 22:43:00.372015 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:00.372153 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:00.372211 (+ 58us) server_negotiation.cc:299] Negotiation successful
0809 22:43:00.372274 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 22:43:00.375873 21081 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa850007cf1a4023bc131bf8d0c92804" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40847 } }
I20260809 22:43:00.376273 21081 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 22:43:00.376415 21081 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa850007cf1a4023bc131bf8d0c92804, State: Initialized, Role: FOLLOWER
I20260809 22:43:00.377133 21081 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa850007cf1a4023bc131bf8d0c92804" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40847 } }
I20260809 22:43:00.377436 21081 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 22:43:00.377590 21081 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 22:43:00.377772 21081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:43:00.380439 21081 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa850007cf1a4023bc131bf8d0c92804" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40847 } }
I20260809 22:43:00.381059 21081 leader_election.cc:302] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa850007cf1a4023bc131bf8d0c92804; no voters:
I20260809 22:43:00.381673 21081 leader_election.cc:288] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 22:43:00.381847 21091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:43:00.382594 21091 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [term 1 LEADER]: Becoming Leader. State: Replica: aa850007cf1a4023bc131bf8d0c92804, State: Running, Role: LEADER
I20260809 22:43:00.383179 21091 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa850007cf1a4023bc131bf8d0c92804" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40847 } }
I20260809 22:43:00.383862 21081 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:43:00.386837 21092 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa850007cf1a4023bc131bf8d0c92804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa850007cf1a4023bc131bf8d0c92804" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40847 } } }
I20260809 22:43:00.387292 21092 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:00.387773 21093 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa850007cf1a4023bc131bf8d0c92804. Latest consensus state: current_term: 1 leader_uuid: "aa850007cf1a4023bc131bf8d0c92804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa850007cf1a4023bc131bf8d0c92804" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 40847 } } }
I20260809 22:43:00.388193 21093 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:00.389899 21098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 22:43:00.396692 21098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:43:00.407321 21098 catalog_manager.cc:1347] Generated new cluster ID: a9e2c1f3e53c43acba55a206635dc171
I20260809 22:43:00.407534 21098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:43:00.433023 21098 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:43:00.433708 21098 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:43:00.442957 21098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa850007cf1a4023bc131bf8d0c92804: Generated new TSK 0
I20260809 22:43:00.444244 21098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 22:43:00.459928 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21009
W20260809 22:43:00.461372 21076 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49937
W20260809 22:43:00.462255 29645 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21009: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 22:43:00.462435 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 21009
2026-08-09T22:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2203 ms)
[----------] 18 tests from AuthTokenIssuingTest (40505 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:43:00 dist-test-slave-rgb1 krb5kdc[21113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:43:00 dist-test-slave-rgb1 krb5kdc[21113](info): set up 2 sockets
Aug 09 22:43:00 dist-test-slave-rgb1 krb5kdc[21113](info): commencing operation
krb5kdc: starting...
W20260809 22:43:01.728894 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:43:01 dist-test-slave-rgb1 krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:43:02.110183 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:44473
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:44473,127.28.243.125:41583,127.28.243.124:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:43:02.474933 21128 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 22:43:02.475502 21128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:43:02.475651 21128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:43:02.475796 21128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:43:02.483434 21128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:43:02.483579 21128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:43:02.483692 21128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:43:02.483785 21128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:43:02.483980 21128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:43:02.494773 21128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:44473,127.28.243.125:41583,127.28.243.124:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:44473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:43:02.496206 21128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:43:02.497637 21128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:43:02.510582 21133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:02.510656 21136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:02.510656 21134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:43:02.511723 21128 server_base.cc:1034] running on GCE node
I20260809 22:43:02.512681 21128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:43:02.514189 21128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:43:02.515498 21128 hybrid_clock.cc:648] HybridClock initialized: now 1786315382515419 us; error 44 us; skew 500 ppm
I20260809 22:43:02.515926 21128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:43:02 dist-test-slave-rgb1 krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315382, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:43:02.525329 21128 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:43:02.527025 21128 webserver.cc:460] Webserver started at http://127.28.243.126:33837/ using document root <none> and password file <none>
I20260809 22:43:02.527753 21128 fs_manager.cc:359] Metadata directory not provided
I20260809 22:43:02.527869 21128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:43:02.528314 21128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:43:02.532078 21128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "fc249b43c41b4324b2927adf07b94c06"
format_stamp: "Formatted at 2026-08-09 22:43:02 on dist-test-slave-rgb1"
server_key: "d08152dc884f4ff513484e4cec5d6cdd"
server_key_iv: "b7e40edd96fe00c45cdf32224ca61263"
server_key_version: "encryptionkey@0"
I20260809 22:43:02.533063 21128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "fc249b43c41b4324b2927adf07b94c06"
format_stamp: "Formatted at 2026-08-09 22:43:02 on dist-test-slave-rgb1"
server_key: "d08152dc884f4ff513484e4cec5d6cdd"
server_key_iv: "b7e40edd96fe00c45cdf32224ca61263"
server_key_version: "encryptionkey@0"
I20260809 22:43:02.539332 21128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:43:02.543689 21143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:43:02.545614 21128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:43:02.545876 21128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "fc249b43c41b4324b2927adf07b94c06"
format_stamp: "Formatted at 2026-08-09 22:43:02 on dist-test-slave-rgb1"
server_key: "d08152dc884f4ff513484e4cec5d6cdd"
server_key_iv: "b7e40edd96fe00c45cdf32224ca61263"
server_key_version: "encryptionkey@0"
I20260809 22:43:02.546241 21128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:43:02.562333 21128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:43:02.567374 21128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:43:02.567660 21128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:43:02.589291 21128 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:44473
I20260809 22:43:02.589298 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:44473 every 8 connection(s)
I20260809 22:43:02.590837 21128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:43:02.594695 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:43:02.597836 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 21128
I20260809 22:43:02.598079 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:43:02.598934 29645 external_mini_cluster.cc:1428] Setting key faab78f6a26565df39626466c67746f7
I20260809 22:43:02.604167 21199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:02 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315382, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:43:02.630927 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.608397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:44473 (local address 127.0.0.1:56140)
0809 22:43:02.609400 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.609453 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:02.611634 (+ 2181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:02.612529 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:02.612536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:02.613300 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:02.613663 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:02.613698 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:02.615432 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:02.615437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:02.616522 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:02.616534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:02.617531 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:02.617538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:02.617796 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:02.618592 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:02.618616 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:02.621417 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:02.625672 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:02.625686 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:02.625701 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:02.626190 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:02.626723 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:02.626731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:02.626736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:02.626935 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:02.627926 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:02.627938 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:02.628219 (+ 281us) client_negotiation.cc:769] Sending connection context
0809 22:43:02.628523 (+ 304us) client_negotiation.cc:240] Negotiation successful
0809 22:43:02.629268 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":174,"thread_start_us":496,"threads_started":1}
I20260809 22:43:02.630949 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.608397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56140 (local address 127.28.243.126:44473)
0809 22:43:02.610328 (+ 1931us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:02.610336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:02.610394 (+ 58us) server_negotiation.cc:407] Connection header received
0809 22:43:02.612042 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:02.612048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:02.612291 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:02.612501 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:02.613916 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:02.615274 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:02.616717 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:02.617274 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:02.617494 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:02.621724 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:02.621734 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:02.621748 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:02.621784 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:02.625490 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:02.626381 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:02.626389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:02.626391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:02.626579 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:02.627075 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:02.627081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:02.627091 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:02.627717 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:02.628020 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:02.628739 (+ 719us) server_negotiation.cc:299] Negotiation successful
0809 22:43:02.629114 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":1697,"thread_start_us":1429,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.125@KRBTEST.COM; defaulting to no policy
I20260809 22:43:02.634363 21199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.125@KRBTEST.COM" created.
I20260809 22:43:02.636062 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.635371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39614)
0809 22:43:02.635704 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.635886 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260809 22:43:02.636477 21144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
I20260809 22:43:02.638227 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.637808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39618)
0809 22:43:02.637992 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.638078 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260809 22:43:02.638549 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:41583: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab.
Entry for principal kudu/127.28.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab.
I20260809 22:43:02.677695 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:41583
--webserver_interface=127.28.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab
--principal=kudu/127.28.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:44473,127.28.243.125:41583,127.28.243.124:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:43:02.687974 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } attempt: 1
I20260809 22:43:02.689446 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.688904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39634)
0809 22:43:02.689172 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.689305 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 22:43:02.690630 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.690259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39640)
0809 22:43:02.690416 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.690494 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 22:43:02.690889 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:41583: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
I20260809 22:43:02.759204 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } attempt: 2
I20260809 22:43:02.760725 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.759983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39656)
0809 22:43:02.760306 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.760436 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 22:43:02.762190 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.761554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39662)
0809 22:43:02.761793 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.761912 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 22:43:02.762512 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:41583: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
I20260809 22:43:02.853806 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } attempt: 3
I20260809 22:43:02.855207 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.854573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39678)
0809 22:43:02.854849 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.854991 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 22:43:02.856748 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:02.856173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39694)
0809 22:43:02.856350 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:02.856431 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260809 22:43:02.857029 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:41583: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
I20260809 22:43:03.000319 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } attempt: 4
I20260809 22:43:03.001736 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.001095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39696)
0809 22:43:03.001475 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.001581 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260809 22:43:03.003010 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.002539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39704)
0809 22:43:03.002712 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.002812 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260809 22:43:03.003218 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:41583: Network error: Client connection negotiation failed: client connection to 127.28.243.125:41583: connect: Connection refused (error 111)
W20260809 22:43:03.044140 21208 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 22:43:03.044544 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:43:03.044709 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:43:03.044848 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:43:03.052853 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:43:03.052989 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:43:03.053113 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:43:03.053198 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:43:03.053305 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:43:03.063730 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:44473,127.28.243.125:41583,127.28.243.124:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab
--principal=kudu/127.28.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:43:03.065197 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:43:03.066607 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:43:03.079156 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:03.079304 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:03.079605 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:43:03.080471 21208 server_base.cc:1034] running on GCE node
I20260809 22:43:03.081286 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:43:03.082769 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:43:03.084028 21208 hybrid_clock.cc:648] HybridClock initialized: now 1786315383083931 us; error 64 us; skew 500 ppm
I20260809 22:43:03.084398 21208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:43:03.088101 21208 init.cc:377] Logged in from keytab as kudu/127.28.243.125@KRBTEST.COM (short username kudu)
I20260809 22:43:03.089951 21208 webserver.cc:460] Webserver started at http://127.28.243.125:32797/ using document root <none> and password file <none>
I20260809 22:43:03.090724 21208 fs_manager.cc:359] Metadata directory not provided
I20260809 22:43:03.090905 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:43:03.091284 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:43:03.094519 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data/instance:
uuid: "0b4345c5d90c4930aa5f8a81e4318f27"
format_stamp: "Formatted at 2026-08-09 22:43:03 on dist-test-slave-rgb1"
server_key: "2dd37d0c4d41fe26ff56673829d00478"
server_key_iv: "528f1298ba79ae00a738177b6ad5919f"
server_key_version: "encryptionkey@0"
I20260809 22:43:03.095343 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/wal/instance:
uuid: "0b4345c5d90c4930aa5f8a81e4318f27"
format_stamp: "Formatted at 2026-08-09 22:43:03 on dist-test-slave-rgb1"
server_key: "2dd37d0c4d41fe26ff56673829d00478"
server_key_iv: "528f1298ba79ae00a738177b6ad5919f"
server_key_version: "encryptionkey@0"
I20260809 22:43:03.101064 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 22:43:03.105058 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:43:03.106618 21208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 22:43:03.106812 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/wal
uuid: "0b4345c5d90c4930aa5f8a81e4318f27"
format_stamp: "Formatted at 2026-08-09 22:43:03 on dist-test-slave-rgb1"
server_key: "2dd37d0c4d41fe26ff56673829d00478"
server_key_iv: "528f1298ba79ae00a738177b6ad5919f"
server_key_version: "encryptionkey@0"
I20260809 22:43:03.107069 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:43:03.127307 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:43:03.132452 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:43:03.132893 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:43:03.154526 21208 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.125:41583
I20260809 22:43:03.154618 21277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.125:41583 every 8 connection(s)
I20260809 22:43:03.155977 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
I20260809 22:43:03.160177 21278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:43:03.164026 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 21208
I20260809 22:43:03.164248 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-1/wal/instance
I20260809 22:43:03.164863 29645 external_mini_cluster.cc:1428] Setting key 07f95726676bd40cd57c4d1203fa2e52
I20260809 22:43:03.168872 21278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:43:03.195818 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.172439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.28.243.126:44473)
0809 22:43:03.173057 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.173064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.174239 (+ 1175us) server_negotiation.cc:407] Connection header received
0809 22:43:03.177339 (+ 3100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.177344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.177424 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.177536 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.179880 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.181262 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.182961 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.183739 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.183921 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.187645 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.187654 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.187663 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.187728 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.191883 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.192860 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.192869 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.192875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.193041 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.193512 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.193519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.193525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.193999 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.194168 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.194997 (+ 829us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.195283 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":312,"threads_started":1}
I20260809 22:43:03.196652 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.173175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:44473 (local address 127.0.0.1:56156)
0809 22:43:03.174032 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.174092 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:03.176962 (+ 2870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:03.177836 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:03.177859 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:03.178696 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:03.179601 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.179629 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.181459 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.181463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.182795 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.182808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.183990 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.183995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.184251 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.185073 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:03.185097 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:03.187370 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:03.192091 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.192099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.192114 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.192726 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.193165 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.193169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.193171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.193379 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.194173 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:03.194180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:03.194584 (+ 404us) client_negotiation.cc:769] Sending connection context
0809 22:43:03.194839 (+ 255us) client_negotiation.cc:240] Negotiation successful
0809 22:43:03.195269 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":317,"threads_started":1}
I20260809 22:43:03.198614 21278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.243.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.124@KRBTEST.COM" created.
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:43:03.215392 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.199501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39714)
0809 22:43:03.199694 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.199717 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:03.199952 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:03.200854 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:03.200924 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:03.201310 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:03.201457 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.201466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.203069 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.203073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.204359 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.204391 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.205386 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.205392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.205610 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.206307 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:03.206327 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:03.208746 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:03.212615 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.212622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.212627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.213041 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.213447 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.213455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.213459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.213620 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.214263 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:03.214270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:03.214447 (+ 177us) client_negotiation.cc:769] Sending connection context
0809 22:43:03.214602 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 22:43:03.214915 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 22:43:03.215430 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.199572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.28.243.125:41583)
0809 22:43:03.200267 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.200277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.200314 (+ 37us) server_negotiation.cc:407] Connection header received
0809 22:43:03.200381 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.200392 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.200680 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.200841 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.201666 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.202908 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.204522 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.205216 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.205383 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.208989 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.209003 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.209018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.209050 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.212450 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.213186 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.213196 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.213205 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.213332 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.213729 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.213738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.213743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.214130 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.214377 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.214775 (+ 398us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.215017 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":299,"threads_started":1}
I20260809 22:43:03.218628 21278 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:43:03.220093 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.219586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33338)
0809 22:43:03.219775 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.219914 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260809 22:43:03.220402 21227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.222108 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.221734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33354)
0809 22:43:03.221907 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.222007 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 22:43:03.222344 21278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab.
Entry for principal kudu/127.28.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab.
I20260809 22:43:03.249310 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:33813
--webserver_interface=127.28.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab
--principal=kudu/127.28.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:44473,127.28.243.125:41583,127.28.243.124:33813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:43:03.271667 21278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 1
I20260809 22:43:03.273213 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.272485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33368)
0809 22:43:03.272794 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.272987 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260809 22:43:03.274485 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.273989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33376)
0809 22:43:03.274208 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.274316 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 22:43:03.274751 21278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.302517 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } attempt: 5
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315382, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:43:03.318758 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.303426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39720)
0809 22:43:03.303665 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.303689 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:03.303848 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:03.304249 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:03.304257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:03.304765 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:03.304959 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.304978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.306652 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.306657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.307735 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.307750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.308804 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.308808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.309030 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.309706 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:03.309725 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:03.312013 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:03.316300 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.316304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.316307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.316764 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.317191 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.317194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.317197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.317341 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.318015 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:03.318018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:03.318145 (+ 127us) client_negotiation.cc:769] Sending connection context
0809 22:43:03.318266 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 22:43:03.318506 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260809 22:43:03.318943 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.303666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.28.243.125:41583)
0809 22:43:03.303880 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.303890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.303917 (+ 27us) server_negotiation.cc:407] Connection header received
0809 22:43:03.303987 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.303993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.304087 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.304206 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.305236 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.306469 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.307906 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.308636 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.308780 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.312232 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.312241 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.312247 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.312299 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.316111 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.316882 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.316887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.316891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.317078 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.317455 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.317459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.317463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.317840 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.317997 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.318398 (+ 401us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.318656 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 22:43:03.320182 21199 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:43:03.321409 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.320988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33386)
0809 22:43:03.321164 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.321261 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 22:43:03.322474 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.322081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33394)
0809 22:43:03.322244 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.322343 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 22:43:03.322691 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.343001 21278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 2
I20260809 22:43:03.344431 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.343777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33400)
0809 22:43:03.344004 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.344209 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 22:43:03.345657 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.345252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33416)
0809 22:43:03.345411 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.345512 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 22:43:03.346006 21278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.373945 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 1
I20260809 22:43:03.375463 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.374772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33428)
0809 22:43:03.375070 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.375181 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260809 22:43:03.376832 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.376387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33436)
0809 22:43:03.376632 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.376713 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260809 22:43:03.377063 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.437299 21278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 3
I20260809 22:43:03.438658 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.438150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33452)
0809 22:43:03.438389 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.438496 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260809 22:43:03.440070 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.439434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33454)
0809 22:43:03.439736 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.439822 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
W20260809 22:43:03.440366 21278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.445282 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 2
I20260809 22:43:03.446475 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.445954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33456)
0809 22:43:03.446212 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.446316 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 22:43:03.447579 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.447175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33464)
0809 22:43:03.447338 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.447430 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 22:43:03.447853 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.554117 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 3
I20260809 22:43:03.555532 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.554906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33480)
0809 22:43:03.555161 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.555285 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260809 22:43:03.556869 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.556355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33490)
0809 22:43:03.556603 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.556707 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260809 22:43:03.557135 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.583617 21278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 4
I20260809 22:43:03.584915 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.584341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33506)
0809 22:43:03.584667 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.584752 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 22:43:03.586040 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.585670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33522)
0809 22:43:03.585840 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.585922 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 22:43:03.586275 21278 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
W20260809 22:43:03.628146 21286 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 22:43:03.628754 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:43:03.628865 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:43:03.628999 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:43:03.636654 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:43:03.636776 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:43:03.636861 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:43:03.636953 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:43:03.637060 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:43:03.647523 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:44473,127.28.243.125:41583,127.28.243.124:33813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab
--principal=kudu/127.28.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:33813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:43:03.649029 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:43:03.650542 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:43:03.663185 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:03.663385 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:03.663194 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:43:03.670524 21286 server_base.cc:1034] running on GCE node
I20260809 22:43:03.671554 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:43:03.673365 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:43:03.674683 21286 hybrid_clock.cc:648] HybridClock initialized: now 1786315383674578 us; error 64 us; skew 500 ppm
I20260809 22:43:03.675145 21286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:43:03.678750 21286 init.cc:377] Logged in from keytab as kudu/127.28.243.124@KRBTEST.COM (short username kudu)
I20260809 22:43:03.680434 21286 webserver.cc:460] Webserver started at http://127.28.243.124:44413/ using document root <none> and password file <none>
I20260809 22:43:03.681252 21286 fs_manager.cc:359] Metadata directory not provided
I20260809 22:43:03.681387 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:43:03.681716 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:43:03.685021 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data/instance:
uuid: "bd43b0de23f540e8b3a3bb02ea4994f4"
format_stamp: "Formatted at 2026-08-09 22:43:03 on dist-test-slave-rgb1"
server_key: "b3f939735be324cdf295a7104c7f6d64"
server_key_iv: "092cf5e349efcbee8d2fe5f6f3cbb3f0"
server_key_version: "encryptionkey@0"
I20260809 22:43:03.685896 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/wal/instance:
uuid: "bd43b0de23f540e8b3a3bb02ea4994f4"
format_stamp: "Formatted at 2026-08-09 22:43:03 on dist-test-slave-rgb1"
server_key: "b3f939735be324cdf295a7104c7f6d64"
server_key_iv: "092cf5e349efcbee8d2fe5f6f3cbb3f0"
server_key_version: "encryptionkey@0"
I20260809 22:43:03.691617 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 22:43:03.695722 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:43:03.697325 21286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 22:43:03.697528 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/wal
uuid: "bd43b0de23f540e8b3a3bb02ea4994f4"
format_stamp: "Formatted at 2026-08-09 22:43:03 on dist-test-slave-rgb1"
server_key: "b3f939735be324cdf295a7104c7f6d64"
server_key_iv: "092cf5e349efcbee8d2fe5f6f3cbb3f0"
server_key_version: "encryptionkey@0"
I20260809 22:43:03.697764 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:43:03.725853 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:43:03.730763 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:43:03.731072 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:43:03.734411 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 4
I20260809 22:43:03.735484 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.735052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33524)
0809 22:43:03.735245 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.735339 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 22:43:03.737130 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.736366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33528)
0809 22:43:03.736800 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.736938 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264}
W20260809 22:43:03.737512 21199 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:33813: Network error: Client connection negotiation failed: client connection to 127.28.243.124:33813: connect: Connection refused (error 111)
I20260809 22:43:03.751739 21286 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.124:33813
I20260809 22:43:03.751747 21355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.124:33813 every 8 connection(s)
I20260809 22:43:03.753217 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
I20260809 22:43:03.756507 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 21286
I20260809 22:43:03.756749 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786314994965430-29645-0/minicluster-data/master-2/wal/instance
I20260809 22:43:03.757318 29645 external_mini_cluster.cc:1428] Setting key 99d3135971c90ee7d8bf8d3a6655474e
I20260809 22:43:03.757905 21356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:43:03.768895 21356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:43:03.784901 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.759483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.28.243.124:33813)
0809 22:43:03.760328 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.760340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.760382 (+ 42us) server_negotiation.cc:407] Connection header received
0809 22:43:03.762880 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.762903 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.763340 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.763682 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.765913 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.767324 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.769475 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.770621 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.770771 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.776378 (+ 5607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.776386 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.776398 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.776437 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.780503 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.781347 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.781357 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.781367 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.781576 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.781974 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.781978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.781991 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.782640 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.782981 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.783390 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.783764 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":384,"threads_started":1}
I20260809 22:43:03.789853 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.772687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.28.243.126:44473)
0809 22:43:03.773371 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.773381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.774195 (+ 814us) server_negotiation.cc:407] Connection header received
0809 22:43:03.774439 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.774444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.774511 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.774611 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.776089 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.776920 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.779158 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.779595 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.779704 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.783231 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.783237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.783242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.783287 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.786266 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.787195 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.787202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.787217 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.787428 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.787926 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.787932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.787936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.788356 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.788614 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.789157 (+ 543us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.789384 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":316,"threads_started":1}
I20260809 22:43:03.789835 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.772775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:44473 (local address 127.0.0.1:56168)
0809 22:43:03.774015 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.774062 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:03.774267 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:03.774714 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:03.774725 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:03.775553 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:03.775842 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.775852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.777100 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.777109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.778883 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.778901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.779800 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.779806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.780018 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.780716 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:03.780742 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:03.783024 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:03.786394 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.786405 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.786413 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.787020 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.787611 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.787616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.787619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.787796 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.788533 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:03.788541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:03.788822 (+ 281us) client_negotiation.cc:769] Sending connection context
0809 22:43:03.789023 (+ 201us) client_negotiation.cc:240] Negotiation successful
0809 22:43:03.789382 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":639,"threads_started":1}
I20260809 22:43:03.791353 21356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:43:03.806402 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.788345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.28.243.126:44473)
0809 22:43:03.789119 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.789124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.789144 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:43:03.789185 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.789189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.789244 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.789372 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.791096 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.792227 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.793993 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.794488 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.794623 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.800196 (+ 5573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.800201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.800205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.800259 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.803577 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.804277 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.804283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.804286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.804436 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.804860 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.804865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.804873 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.805164 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.805458 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.805783 (+ 325us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.806017 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":432,"threads_started":1}
I20260809 22:43:03.807025 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.792170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:41583 (local address 127.0.0.1:39724)
0809 22:43:03.792464 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.792483 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:03.792625 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:03.793057 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:03.793060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:03.793427 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:03.793590 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.793599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.794931 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.794936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.795989 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.796003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.797202 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.797207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.797358 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.797974 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:03.797992 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:03.800433 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:03.804189 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.804195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.804199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.804693 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.805119 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.805124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.805128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.805291 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.805876 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:03.805882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:03.806068 (+ 186us) client_negotiation.cc:769] Sending connection context
0809 22:43:03.806236 (+ 168us) client_negotiation.cc:240] Negotiation successful
0809 22:43:03.806626 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260809 22:43:03.807986 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.792169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.28.243.125:41583)
0809 22:43:03.792353 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.792360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.792553 (+ 193us) server_negotiation.cc:407] Connection header received
0809 22:43:03.792739 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.792747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.792850 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.792955 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.793771 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.794754 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.796158 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.797025 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.797195 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.800673 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.800678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.800682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.800718 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.804036 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.804818 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.804826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.804830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.804992 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.805410 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.805416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.805419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.805734 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.805914 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.807375 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.807649 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 22:43:03.809497 21356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
I20260809 22:43:03.824298 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.808500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.28.243.125:41583)
0809 22:43:03.808710 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.808718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.808735 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:43:03.808780 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.808785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.808862 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.808972 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.810598 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.811824 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.813211 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.813653 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.813795 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.818477 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.818482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.818485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.818519 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.821426 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.822191 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.822196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.822198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.822347 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.822752 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.822757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.822759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.823036 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.823339 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.823630 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.823890 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:43:03.824714 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.810304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33558)
0809 22:43:03.810640 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.810662 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:03.810787 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:03.811270 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:03.811276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:03.811836 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:03.812043 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.812053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.813187 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.813196 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.814113 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.814125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.814873 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.814879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.815142 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.815736 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:03.815754 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:03.818458 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:03.821971 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.821977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.821981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.822378 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.822836 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.822842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.822846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.822993 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.823550 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:03.823556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:03.823707 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 22:43:03.823873 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 22:43:03.824225 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260809 22:43:03.824826 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.810305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.28.243.124:33813)
0809 22:43:03.810520 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.810528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.810886 (+ 358us) server_negotiation.cc:407] Connection header received
0809 22:43:03.810962 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.810968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.811104 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.811212 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.812192 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.813060 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.814257 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.814690 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.814852 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.818655 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.818660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.818663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.818693 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.821803 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.822511 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.822520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.822525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.822710 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.823101 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.823109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.823113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.823429 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.823591 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.823988 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.824422 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":54}
I20260809 22:43:03.829555 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:43:03.830713 21356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: Bootstrap starting.
I20260809 22:43:03.834085 21356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: Neither blocks nor log segments found. Creating new log.
I20260809 22:43:03.832350 21172 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fc249b43c41b4324b2927adf07b94c06"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56172
I20260809 22:43:03.835209 21356 log.cc:824] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: Log is configured to *not* fsync() on all Append() calls
I20260809 22:43:03.838164 21356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: No bootstrap required, opened a new log
I20260809 22:43:03.846112 21356 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:03.846444 21356 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd43b0de23f540e8b3a3bb02ea4994f4, State: Initialized, Role: FOLLOWER
I20260809 22:43:03.846925 21356 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:03.847995 21364 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:03.848407 21364 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:03.849267 21356 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 22:43:03.859582 21376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:43:03.859743 21376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:43:03.857849 21286 master_runner.cc:186] Error getting master registration for 127.28.243.126:44473: OK, instance_id { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" instance_seqno: 1786315382582944 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 22:43:03.866096 21286 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 22:43:03.885486 21278 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 5
Aug 09 22:43:03 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315383, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
I20260809 22:43:03.901599 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.886311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33568)
0809 22:43:03.886550 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:03.886583 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:03.886723 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:03.887128 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:03.887132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:03.887570 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:03.887729 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.887738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.888886 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.888891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.889768 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:03.889780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.890867 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.890874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:03.891175 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.892019 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:03.892047 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:03.895023 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:03.898276 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.898282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.898286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.898705 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.899208 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:03.899216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:03.899219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:03.899594 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:03.900332 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:03.900338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:03.900534 (+ 196us) client_negotiation.cc:769] Sending connection context
0809 22:43:03.900772 (+ 238us) client_negotiation.cc:240] Negotiation successful
0809 22:43:03.901130 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 22:43:03.901607 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:03.886428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.28.243.124:33813)
0809 22:43:03.886649 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:03.886656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:03.886679 (+ 23us) server_negotiation.cc:407] Connection header received
0809 22:43:03.886905 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:03.886910 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:03.886986 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:03.887102 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:03.887881 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.888733 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.889914 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:03.890669 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:03.890800 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:03.895226 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:03.895231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:03.895234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:03.895268 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:03.898119 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.898840 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.898847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.898851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.898996 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:03.899754 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:03.899761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:03.899765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:03.900179 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:03.900431 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:03.900869 (+ 438us) server_negotiation.cc:299] Negotiation successful
0809 22:43:03.901141 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 22:43:03.908010 21278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: Bootstrap starting.
I20260809 22:43:03.912698 21278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: Neither blocks nor log segments found. Creating new log.
I20260809 22:43:03.914186 21278 log.cc:824] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: Log is configured to *not* fsync() on all Append() calls
I20260809 22:43:03.917846 21278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: No bootstrap required, opened a new log
I20260809 22:43:03.930063 21278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:03.930581 21278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4345c5d90c4930aa5f8a81e4318f27, State: Initialized, Role: FOLLOWER
I20260809 22:43:03.931113 21278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:03.932276 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:03.932837 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:03.933650 21278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 22:43:03.946321 21393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:43:03.946516 21393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:43:03.943694 21208 master_runner.cc:186] Error getting master registration for 127.28.243.126:44473: OK, instance_id { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" instance_seqno: 1786315382582944 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 22:43:03.955212 21208 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 22:43:03.983543 21307 tablet.cc:2369] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:43:04.014815 21199 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } attempt: 5
Aug 09 22:43:04 dist-test-slave-rgb1 krb5kdc[21113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315382, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
I20260809 22:43:04.035418 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:04.015620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:33813 (local address 127.0.0.1:33572)
0809 22:43:04.015870 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:04.015897 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:04.016060 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:04.016507 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:04.016514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:04.017149 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:04.017427 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:04.017443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.019159 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.019165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:04.020915 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:04.020958 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.022103 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.022111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:04.022337 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:04.023182 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:04.023211 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:04.026739 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:04.031833 (+ 5094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:04.031840 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:04.031904 (+ 64us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:04.032521 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:04.033051 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:04.033057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:04.033061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:04.033241 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:04.034119 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:04.034125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:04.034327 (+ 202us) client_negotiation.cc:769] Sending connection context
0809 22:43:04.034564 (+ 237us) client_negotiation.cc:240] Negotiation successful
0809 22:43:04.034944 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 22:43:04.035434 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:04.015964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.28.243.124:33813)
0809 22:43:04.016161 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.016167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.016189 (+ 22us) server_negotiation.cc:407] Connection header received
0809 22:43:04.016256 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.016261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.016344 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.016461 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.017670 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.019008 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.021157 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.021906 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.022058 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:04.027006 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:04.027050 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:04.027057 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:04.027119 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:04.031638 (+ 4519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.032704 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.032711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.032715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.032905 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.033382 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.033388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.033392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.033856 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:04.034049 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:04.034726 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 22:43:04.035001 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 22:43:04.042807 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06: Bootstrap starting.
I20260809 22:43:04.047847 21199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06: Neither blocks nor log segments found. Creating new log.
I20260809 22:43:04.049801 21199 log.cc:824] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06: Log is configured to *not* fsync() on all Append() calls
I20260809 22:43:04.061022 21199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06: No bootstrap required, opened a new log
I20260809 22:43:04.070480 21199 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:04.071230 21199 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc249b43c41b4324b2927adf07b94c06, State: Initialized, Role: FOLLOWER
I20260809 22:43:04.071852 21199 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:04.073506 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.074432 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:04.075146 21199 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:43:04.075241 21172 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 22:43:04.075460 21172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:43:04.079264 21172 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:04.082283 21329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc249b43c41b4324b2927adf07b94c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4"
I20260809 22:43:04.082347 21252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc249b43c41b4324b2927adf07b94c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0b4345c5d90c4930aa5f8a81e4318f27"
I20260809 22:43:04.082837 21329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:43:04.082890 21252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:43:04.086510 21329 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc249b43c41b4324b2927adf07b94c06 in term 1.
I20260809 22:43:04.087162 21252 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc249b43c41b4324b2927adf07b94c06 in term 1.
I20260809 22:43:04.087304 21147 leader_election.cc:302] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd43b0de23f540e8b3a3bb02ea4994f4, fc249b43c41b4324b2927adf07b94c06; no voters:
I20260809 22:43:04.089028 21172 leader_election.cc:288] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [CANDIDATE]: Term 1 election: Requested vote from peers 0b4345c5d90c4930aa5f8a81e4318f27 (127.28.243.125:41583), bd43b0de23f540e8b3a3bb02ea4994f4 (127.28.243.124:33813)
I20260809 22:43:04.089267 21399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:43:04.096899 21399 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [term 1 LEADER]: Becoming Leader. State: Replica: fc249b43c41b4324b2927adf07b94c06, State: Running, Role: LEADER
I20260809 22:43:04.097807 21399 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } }
I20260809 22:43:04.109299 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc249b43c41b4324b2927adf07b94c06. Latest consensus state: current_term: 1 leader_uuid: "fc249b43c41b4324b2927adf07b94c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.109809 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:04.122295 21407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260809 22:43:04.136162 21229 tablet.cc:2369] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:43:04.168119 21407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:43:04.193063 21420 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:43:04.213795 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:04.147274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.28.243.124:33813)
0809 22:43:04.147529 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.147536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.171603 (+ 24067us) server_negotiation.cc:407] Connection header received
0809 22:43:04.172145 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.172152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.172256 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.172368 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.175310 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.176468 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.191199 (+ 14731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.191993 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.192129 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:04.206302 (+ 14173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:04.206308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:04.206314 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:04.206357 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:04.210031 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.210931 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.210936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.210939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.211164 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.211667 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.211673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.211676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.212272 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:04.212516 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:04.213134 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 22:43:04.213387 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:43:04.218235 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:04.130660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.28.243.126:44473)
0809 22:43:04.130937 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.130944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.149969 (+ 19025us) server_negotiation.cc:407] Connection header received
0809 22:43:04.150435 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.150441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.150534 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.150619 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.172153 (+ 21534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.185337 (+ 13184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.187737 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.188545 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.188706 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:04.209227 (+ 20521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:04.209235 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:04.209240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:04.209285 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:04.213196 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.214494 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.214499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.214503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.214657 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.215543 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.215550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.215553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.215922 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:04.216153 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:04.217634 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0809 22:43:04.217884 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:43:04.219070 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:04.132839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.28.243.125:41583)
0809 22:43:04.133039 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.133046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.152802 (+ 19756us) server_negotiation.cc:407] Connection header received
0809 22:43:04.153085 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.153091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.153176 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.153250 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.155860 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.157158 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.182457 (+ 25299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:04.183198 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:04.183310 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:04.203645 (+ 20335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:04.203655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:04.203666 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:04.203714 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:04.208166 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.214095 (+ 5929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.214102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.214106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.214291 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:04.216261 (+ 1970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:04.216267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:04.216272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:04.216739 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:04.217003 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:04.218453 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0809 22:43:04.218728 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:43:04.231940 21329 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [term 1 FOLLOWER]: Refusing update from remote peer fc249b43c41b4324b2927adf07b94c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:43:04.231940 21252 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [term 1 FOLLOWER]: Refusing update from remote peer fc249b43c41b4324b2927adf07b94c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:43:04.233088 21397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:43:04.233853 21399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:43:04.260949 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc249b43c41b4324b2927adf07b94c06. Latest consensus state: current_term: 1 leader_uuid: "fc249b43c41b4324b2927adf07b94c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.261355 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:04.263116 21364 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc249b43c41b4324b2927adf07b94c06. Latest consensus state: current_term: 1 leader_uuid: "fc249b43c41b4324b2927adf07b94c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.263486 21364 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:04.267834 21424 mvcc.cc:205] Tried to move back new op lower bound from 7316747813801037824 to 7316747813282619392. Current Snapshot: MvccSnapshot[applied={T|T < 7316747813801037824}]
I20260809 22:43:04.268451 21421 mvcc.cc:205] Tried to move back new op lower bound from 7316747813801037824 to 7316747813282619392. Current Snapshot: MvccSnapshot[applied={T|T < 7316747813801037824}]
I20260809 22:43:04.280647 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc249b43c41b4324b2927adf07b94c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.281010 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:04.282297 21364 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc249b43c41b4324b2927adf07b94c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.282624 21364 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:04.287185 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc249b43c41b4324b2927adf07b94c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.287523 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:04.287866 21399 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc249b43c41b4324b2927adf07b94c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc249b43c41b4324b2927adf07b94c06" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 44473 } } peers { permanent_uuid: "0b4345c5d90c4930aa5f8a81e4318f27" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 41583 } } peers { permanent_uuid: "bd43b0de23f540e8b3a3bb02ea4994f4" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 33813 } } }
I20260809 22:43:04.288096 21399 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:04.288200 21407 catalog_manager.cc:1347] Generated new cluster ID: 713e3f9822ef456a87c81f0773ae4e66
I20260809 22:43:04.288316 21407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:43:04.341527 21407 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:43:04.344012 21407 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:43:04.387916 21407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc249b43c41b4324b2927adf07b94c06: Generated new TSK 0
I20260809 22:43:04.389381 21407 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 22:43:04.524832 21362 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.512980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.28.243.126:44473)
0809 22:43:04.513168 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.513178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.522102 (+ 8924us) server_negotiation.cc:407] Connection header received
0809 22:43:04.522736 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.522744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.522867 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.522985 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.524446 (+ 1461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56198: BlockingRecv error: recv got EOF from 127.0.0.1:56198 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 22:43:04.525203 21359 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.512501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.28.243.124:33813)
0809 22:43:04.512762 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.512770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.523065 (+ 10295us) server_negotiation.cc:407] Connection header received
0809 22:43:04.523370 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.523397 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.523537 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.523624 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.524947 (+ 1323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33594: BlockingRecv error: recv got EOF from 127.0.0.1:33594 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260809 22:43:04.525431 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.513371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39760 (local address 127.28.243.125:41583)
0809 22:43:04.513622 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.513628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.523345 (+ 9717us) server_negotiation.cc:407] Connection header received
0809 22:43:04.523674 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.523681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.523811 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.523891 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.525154 (+ 1263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39760: BlockingRecv error: recv got EOF from 127.0.0.1:39760 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 22:43:04.534124 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.531673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.28.243.125:41583)
0809 22:43:04.531849 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.531856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.532309 (+ 453us) server_negotiation.cc:407] Connection header received
0809 22:43:04.532553 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.532583 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.532728 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.532860 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.533791 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39768: BlockingRecv error: recv got EOF from 127.0.0.1:39768 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 22:43:04.541570 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.539274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.28.243.125:41583)
0809 22:43:04.539445 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.539451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.539919 (+ 468us) server_negotiation.cc:407] Connection header received
0809 22:43:04.540110 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.540117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.540217 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.540336 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.541330 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39780: BlockingRecv error: recv got EOF from 127.0.0.1:39780 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 22:43:04.553475 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.550866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.28.243.125:41583)
0809 22:43:04.551085 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.551092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.551832 (+ 740us) server_negotiation.cc:407] Connection header received
0809 22:43:04.552045 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.552050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.552172 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.552268 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.553281 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39796: BlockingRecv error: recv got EOF from 127.0.0.1:39796 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 22:43:04.564204 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.561834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.28.243.125:41583)
0809 22:43:04.562059 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.562064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.562485 (+ 421us) server_negotiation.cc:407] Connection header received
0809 22:43:04.562750 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.562757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.562839 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.562936 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.563915 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39798: BlockingRecv error: recv got EOF from 127.0.0.1:39798 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260809 22:43:04.578330 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.576112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.28.243.125:41583)
0809 22:43:04.576313 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.576321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.576657 (+ 336us) server_negotiation.cc:407] Connection header received
0809 22:43:04.576983 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.576988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.577072 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.577156 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.578099 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39814: BlockingRecv error: recv got EOF from 127.0.0.1:39814 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260809 22:43:04.600373 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.598178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.28.243.125:41583)
0809 22:43:04.598443 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.598450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.598826 (+ 376us) server_negotiation.cc:407] Connection header received
0809 22:43:04.599039 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.599045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.599141 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.599255 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.600133 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39822: BlockingRecv error: recv got EOF from 127.0.0.1:39822 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260809 22:43:04.639231 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.636838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.28.243.125:41583)
0809 22:43:04.637075 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.637082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.637646 (+ 564us) server_negotiation.cc:407] Connection header received
0809 22:43:04.637852 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.637858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.637963 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.638092 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.638936 (+ 844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39838: BlockingRecv error: recv got EOF from 127.0.0.1:39838 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 22:43:04.709563 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.707484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.28.243.125:41583)
0809 22:43:04.707782 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.707790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.708161 (+ 371us) server_negotiation.cc:407] Connection header received
0809 22:43:04.708328 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.708334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.708452 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.708633 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.709366 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39852: BlockingRecv error: recv got EOF from 127.0.0.1:39852 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260809 22:43:04.843786 21283 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:04.841656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.28.243.125:41583)
0809 22:43:04.841956 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:04.841963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:04.842321 (+ 358us) server_negotiation.cc:407] Connection header received
0809 22:43:04.842563 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:04.842568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:04.842643 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:04.842754 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:04.843575 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39856: BlockingRecv error: recv got EOF from 127.0.0.1:39856 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260809 22:43:04.862107 21376 catalog_manager.cc:1259] Loaded cluster ID: 713e3f9822ef456a87c81f0773ae4e66
I20260809 22:43:04.862216 21376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: loading cluster ID for follower catalog manager: success
I20260809 22:43:04.864873 21376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: acquiring CA information for follower catalog manager: success
I20260809 22:43:04.866679 21376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd43b0de23f540e8b3a3bb02ea4994f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:43:04.948930 21393 catalog_manager.cc:1259] Loaded cluster ID: 713e3f9822ef456a87c81f0773ae4e66
I20260809 22:43:04.949050 21393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: loading cluster ID for follower catalog manager: success
I20260809 22:43:04.951839 21393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: acquiring CA information for follower catalog manager: success
I20260809 22:43:04.953812 21393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b4345c5d90c4930aa5f8a81e4318f27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:43:05.110733 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:05.104273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.28.243.125:41583)
0809 22:43:05.104643 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:05.104652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:05.104765 (+ 113us) server_negotiation.cc:407] Connection header received
0809 22:43:05.105031 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:05.105036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:05.105203 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:05.105336 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:05.105881 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.106653 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.108259 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.108751 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.108885 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:05.109411 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:05.109991 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:05.110247 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:05.110351 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 22:43:05.110445 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260809 22:43:05.125281 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:05.117586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.28.243.124:33813)
0809 22:43:05.118839 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:05.118848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:05.118866 (+ 18us) server_negotiation.cc:407] Connection header received
0809 22:43:05.119015 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:05.119019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:05.119231 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:05.119411 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:05.119892 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.120924 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.122490 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.122966 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.123085 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:05.123726 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:05.124362 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:05.124488 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:05.124692 (+ 204us) server_negotiation.cc:299] Negotiation successful
0809 22:43:05.124783 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":1040,"threads_started":1}
I20260809 22:43:05.140955 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:05.133675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.28.243.124:33813)
0809 22:43:05.133875 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:05.133882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:05.134677 (+ 795us) server_negotiation.cc:407] Connection header received
0809 22:43:05.134868 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:05.134876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:05.134966 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:05.135071 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:05.135774 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.136841 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.138968 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.139486 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.139649 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:05.140094 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:05.140232 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:05.140324 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:05.140499 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 22:43:05.140617 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 22:43:05.142020 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:05.133417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.28.243.125:41583)
0809 22:43:05.134073 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:05.134079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:05.134547 (+ 468us) server_negotiation.cc:407] Connection header received
0809 22:43:05.134864 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:05.134871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:05.134952 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:05.135056 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:05.135621 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.136838 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.139686 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.140216 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.140323 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:05.141135 (+ 812us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:05.141269 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:05.141366 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:05.141589 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 22:43:05.141688 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526}
I20260809 22:43:05.143038 21503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:05.132746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.28.243.126:44473)
0809 22:43:05.135303 (+ 2557us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:05.135311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:05.135336 (+ 25us) server_negotiation.cc:407] Connection header received
0809 22:43:05.135468 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:05.135474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:05.135716 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:05.135921 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:05.136407 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.137406 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.139765 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:05.140325 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:05.141054 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:05.141259 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:05.141725 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:05.141890 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:05.142055 (+ 165us) server_negotiation.cc:299] Negotiation successful
0809 22:43:05.142242 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1991,"thread_start_us":1811,"threads_started":1}
I20260809 22:43:05.148818 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21128
I20260809 22:43:05.150275 21194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:43:05.265421 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 21128
W20260809 22:43:05.290323 21305 connection.cc:535] server connection from 127.0.0.1:33572 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260809 22:43:05.290434 21226 connection.cc:535] server connection from 127.0.0.1:39720 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 22:43:05.290999 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21208
I20260809 22:43:05.292302 21273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:43:05.408375 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 21208
I20260809 22:43:05.432039 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21286
I20260809 22:43:05.433362 21351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:43:05.545106 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 21286
2026-08-09T22:43:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5084 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 22:43:05 dist-test-slave-rgb1 krb5kdc[21513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 22:43:05 dist-test-slave-rgb1 krb5kdc[21513](info): set up 2 sockets
Aug 09 22:43:05 dist-test-slave-rgb1 krb5kdc[21513](info): commencing operation
krb5kdc: starting...
W20260809 22:43:06.804095 29645 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/test-user.keytab.
Aug 09 22:43:07 dist-test-slave-rgb1 krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T22:43:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T22:43:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.243.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
Entry for principal kudu/127.28.243.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab.
I20260809 22:43:07.185032 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:32853
--webserver_interface=127.28.243.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:32853,127.28.243.125:33979,127.28.243.124:42693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 22:43:07.564424 21527 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 22:43:07.564944 21527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:43:07.565074 21527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:43:07.565220 21527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:43:07.573258 21527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:43:07.573385 21527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:43:07.573474 21527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:43:07.573577 21527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:43:07.573697 21527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:43:07.584463 21527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:32853,127.28.243.125:33979,127.28.243.124:42693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.126.keytab
--principal=kudu/127.28.243.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.126:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.243.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:43:07.586213 21527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:43:07.587666 21527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:43:07.601344 21532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:07.601274 21535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:07.601303 21533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:43:07.602399 21527 server_base.cc:1034] running on GCE node
I20260809 22:43:07.603399 21527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:43:07.605113 21527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:43:07.606422 21527 hybrid_clock.cc:648] HybridClock initialized: now 1786315387606335 us; error 55 us; skew 500 ppm
I20260809 22:43:07.606822 21527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:43:07 dist-test-slave-rgb1 krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:43:07.617714 21527 init.cc:377] Logged in from keytab as kudu/127.28.243.126@KRBTEST.COM (short username kudu)
I20260809 22:43:07.619567 21527 webserver.cc:460] Webserver started at http://127.28.243.126:41861/ using document root <none> and password file <none>
I20260809 22:43:07.620404 21527 fs_manager.cc:359] Metadata directory not provided
I20260809 22:43:07.620548 21527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:43:07.620952 21527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:43:07.624818 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data/instance:
uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9"
format_stamp: "Formatted at 2026-08-09 22:43:07 on dist-test-slave-rgb1"
server_key: "93ec41e80ddad4971ee36a446613a887"
server_key_iv: "d4bea99699ee94d733ebc3eeaa52e821"
server_key_version: "encryptionkey@0"
I20260809 22:43:07.625787 21527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/wal/instance:
uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9"
format_stamp: "Formatted at 2026-08-09 22:43:07 on dist-test-slave-rgb1"
server_key: "93ec41e80ddad4971ee36a446613a887"
server_key_iv: "d4bea99699ee94d733ebc3eeaa52e821"
server_key_version: "encryptionkey@0"
I20260809 22:43:07.631758 21527 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260809 22:43:07.636137 21542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:43:07.637950 21527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 22:43:07.638183 21527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/wal
uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9"
format_stamp: "Formatted at 2026-08-09 22:43:07 on dist-test-slave-rgb1"
server_key: "93ec41e80ddad4971ee36a446613a887"
server_key_iv: "d4bea99699ee94d733ebc3eeaa52e821"
server_key_version: "encryptionkey@0"
I20260809 22:43:07.638451 21527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:43:07.662351 21527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:43:07.667903 21527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:43:07.668349 21527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:43:07.690733 21527 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.126:32853
I20260809 22:43:07.690761 21596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.126:32853 every 8 connection(s)
I20260809 22:43:07.692359 21527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/data/info.pb
I20260809 22:43:07.693760 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 21527
I20260809 22:43:07.693987 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-0/wal/instance
I20260809 22:43:07.694752 29645 external_mini_cluster.cc:1428] Setting key b9c66bc227f0febd34c9406e4c3982ad
I20260809 22:43:07.697963 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:43:07.709475 21598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:07 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.243.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.125@KRBTEST.COM" created.
I20260809 22:43:07.735685 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.713766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:32853 (local address 127.0.0.1:55866)
0809 22:43:07.714631 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.714677 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:07.716967 (+ 2290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:07.717867 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:07.717873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:07.718625 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:07.718967 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:07.718984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:07.720864 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:07.720870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:07.722118 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:07.722132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:07.723227 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:07.723235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:07.723548 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:07.724324 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:07.724350 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:07.727037 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:07.730925 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:07.730939 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:07.730953 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:07.731498 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:07.731971 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:07.731978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:07.731982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:07.732249 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:07.733442 (+ 1193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:07.733450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:07.733677 (+ 227us) client_negotiation.cc:769] Sending connection context
0809 22:43:07.733925 (+ 248us) client_negotiation.cc:240] Negotiation successful
0809 22:43:07.734380 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":4,"thread_start_us":372,"threads_started":1}
I20260809 22:43:07.735801 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.715191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.28.243.126:32853)
0809 22:43:07.715979 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:07.715990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:07.716051 (+ 61us) server_negotiation.cc:407] Connection header received
0809 22:43:07.717352 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:07.717364 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:07.717633 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:07.717845 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:07.719189 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:07.720683 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:07.722282 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:07.722924 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:07.723173 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:07.727301 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:07.727311 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:07.727326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:07.727358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:07.730754 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:07.731664 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:07.731677 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:07.731686 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:07.731845 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:07.732379 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:07.732386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:07.732395 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:07.733255 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:07.733513 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:07.734140 (+ 627us) server_negotiation.cc:299] Negotiation successful
0809 22:43:07.734570 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":552,"thread_start_us":366,"threads_started":1}
I20260809 22:43:07.739674 21598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:43:07.740984 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.740465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41158)
0809 22:43:07.740716 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.740850 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260809 22:43:07.741369 21546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
I20260809 22:43:07.743265 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.742861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41166)
0809 22:43:07.743042 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.743118 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 22:43:07.743472 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:33979: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab.
Entry for principal kudu/127.28.243.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab.
I20260809 22:43:07.778434 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:33979
--webserver_interface=127.28.243.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab
--principal=kudu/127.28.243.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:32853,127.28.243.125:33979,127.28.243.124:42693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:43:07.792841 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } attempt: 1
I20260809 22:43:07.794298 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.793613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41178)
0809 22:43:07.793908 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.794089 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260809 22:43:07.795548 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.795122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41188)
0809 22:43:07.795297 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.795383 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260809 22:43:07.795857 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:33979: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
I20260809 22:43:07.864202 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } attempt: 2
I20260809 22:43:07.865613 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.865100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41198)
0809 22:43:07.865356 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.865454 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 22:43:07.866796 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.866419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41200)
0809 22:43:07.866581 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.866656 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 22:43:07.867096 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:33979: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
I20260809 22:43:07.958483 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } attempt: 3
I20260809 22:43:07.959965 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.959326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41212)
0809 22:43:07.959627 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.959776 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 22:43:07.961454 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:07.960961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41228)
0809 22:43:07.961151 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:07.961263 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260809 22:43:07.961742 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:33979: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
I20260809 22:43:08.105073 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } attempt: 4
I20260809 22:43:08.106670 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.105908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41240)
0809 22:43:08.106322 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.106455 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260809 22:43:08.107784 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.107406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41246)
0809 22:43:08.107579 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.107662 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260809 22:43:08.108134 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.125:33979: Network error: Client connection negotiation failed: client connection to 127.28.243.125:33979: connect: Connection refused (error 111)
W20260809 22:43:08.152627 21605 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 22:43:08.153085 21605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:43:08.153213 21605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:43:08.153354 21605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:43:08.161243 21605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:43:08.161379 21605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:43:08.161521 21605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:43:08.161612 21605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:43:08.161727 21605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:43:08.172458 21605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:32853,127.28.243.125:33979,127.28.243.124:42693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.125.keytab
--principal=kudu/127.28.243.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.125:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.243.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:43:08.173954 21605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:43:08.175321 21605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:43:08.188369 21610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:08.188287 21611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:08.188334 21613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:43:08.189483 21605 server_base.cc:1034] running on GCE node
I20260809 22:43:08.190407 21605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:43:08.191944 21605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:43:08.193216 21605 hybrid_clock.cc:648] HybridClock initialized: now 1786315388193154 us; error 47 us; skew 500 ppm
I20260809 22:43:08.193614 21605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:43:08.197373 21605 init.cc:377] Logged in from keytab as kudu/127.28.243.125@KRBTEST.COM (short username kudu)
I20260809 22:43:08.199028 21605 webserver.cc:460] Webserver started at http://127.28.243.125:41447/ using document root <none> and password file <none>
I20260809 22:43:08.199797 21605 fs_manager.cc:359] Metadata directory not provided
I20260809 22:43:08.199898 21605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:43:08.200188 21605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:43:08.203616 21605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data/instance:
uuid: "5f1f8101f8bd4fdda62460217fe6222c"
format_stamp: "Formatted at 2026-08-09 22:43:08 on dist-test-slave-rgb1"
server_key: "84cbd7e578b4a7c1c8b7cbd15ccaf5a0"
server_key_iv: "106d7796cff6b18041a1ce8fed457f31"
server_key_version: "encryptionkey@0"
I20260809 22:43:08.204725 21605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/wal/instance:
uuid: "5f1f8101f8bd4fdda62460217fe6222c"
format_stamp: "Formatted at 2026-08-09 22:43:08 on dist-test-slave-rgb1"
server_key: "84cbd7e578b4a7c1c8b7cbd15ccaf5a0"
server_key_iv: "106d7796cff6b18041a1ce8fed457f31"
server_key_version: "encryptionkey@0"
I20260809 22:43:08.210798 21605 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260809 22:43:08.214972 21620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:43:08.216549 21605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 22:43:08.216812 21605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/wal
uuid: "5f1f8101f8bd4fdda62460217fe6222c"
format_stamp: "Formatted at 2026-08-09 22:43:08 on dist-test-slave-rgb1"
server_key: "84cbd7e578b4a7c1c8b7cbd15ccaf5a0"
server_key_iv: "106d7796cff6b18041a1ce8fed457f31"
server_key_version: "encryptionkey@0"
I20260809 22:43:08.217041 21605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:43:08.242148 21605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:43:08.247226 21605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:43:08.247474 21605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:43:08.268996 21605 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.125:33979
I20260809 22:43:08.269016 21676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.125:33979 every 8 connection(s)
I20260809 22:43:08.270566 21605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/data/info.pb
I20260809 22:43:08.275264 21677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:43:08.275930 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 21605
I20260809 22:43:08.276197 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-1/wal/instance
I20260809 22:43:08.276971 29645 external_mini_cluster.cc:1428] Setting key aee1fdcf529e8debe29de1fb76e0df8a
I20260809 22:43:08.287966 21677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:43:08.311685 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.292258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.28.243.126:32853)
0809 22:43:08.292950 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.292955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.294218 (+ 1263us) server_negotiation.cc:407] Connection header received
0809 22:43:08.296315 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.296322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.296433 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.296562 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.298256 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.299047 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.300730 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.301186 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.301334 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.305153 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.305159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.305162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.305203 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.308256 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.309163 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.309168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.309171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.309298 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.309753 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.309766 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.309768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.310076 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.310242 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.311051 (+ 809us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.311286 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":392,"threads_started":1}
WARNING: no policy specified for kudu/127.28.243.124@KRBTEST.COM; defaulting to no policy
I20260809 22:43:08.312877 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.293045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:32853 (local address 127.0.0.1:55874)
0809 22:43:08.294002 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.294076 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:08.296011 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:08.296855 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:08.296864 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:08.297441 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:08.298023 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.298042 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.299232 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.299236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.300476 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.300495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.301449 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.301457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.301703 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.302486 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:08.302510 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:08.304881 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:08.308431 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.308441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.308460 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.309018 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.309422 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.309427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.309430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.309626 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.310215 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:08.310224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:08.310665 (+ 441us) client_negotiation.cc:769] Sending connection context
0809 22:43:08.310949 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 22:43:08.311524 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":372,"threads_started":1}
I20260809 22:43:08.315021 21677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.243.124@KRBTEST.COM" created.
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:43:08.334848 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.315781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41248)
0809 22:43:08.315991 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.316075 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:08.316176 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:08.317644 (+ 1468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:08.317648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:08.318099 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:08.318241 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.318250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.320334 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.320338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.321562 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.321576 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.323081 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.323087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.323257 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.324001 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:08.324020 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:08.326263 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:08.331345 (+ 5082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.331350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.331353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.331932 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.332498 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.332506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.332513 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.332744 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.333565 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:08.333574 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:08.333767 (+ 193us) client_negotiation.cc:769] Sending connection context
0809 22:43:08.333938 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 22:43:08.334291 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260809 22:43:08.334928 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.316254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.28.243.125:33979)
0809 22:43:08.317063 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.317074 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.317109 (+ 35us) server_negotiation.cc:407] Connection header received
0809 22:43:08.317171 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.317183 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.317441 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.317609 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.318438 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.320172 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.321748 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.322845 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.323015 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.326534 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.326554 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.326577 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.326627 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.331144 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.332135 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.332153 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.332156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.332357 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.332882 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.332890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.332896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.333390 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.333689 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.334114 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.334441 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":588,"thread_start_us":423,"threads_started":1}
I20260809 22:43:08.337994 21677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:43:08.339329 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.338811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44704)
0809 22:43:08.339046 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.339166 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260809 22:43:08.339591 21624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.341315 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.340938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44712)
0809 22:43:08.341126 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.341211 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260809 22:43:08.341532 21677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.243.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab.
Entry for principal kudu/127.28.243.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab.
I20260809 22:43:08.361593 29645 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
/tmp/dist-test-taskmuNQXR/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:42693
--webserver_interface=127.28.243.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab
--principal=kudu/127.28.243.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.243.84:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.243.126:32853,127.28.243.125:33979,127.28.243.124:42693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 22:43:08.390846 21677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 1
I20260809 22:43:08.392048 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.391585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44724)
0809 22:43:08.391779 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.391891 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 22:43:08.393090 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.392729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44736)
0809 22:43:08.392914 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.392991 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 22:43:08.393303 21677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.407424 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } attempt: 5
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:43:08.423976 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.408443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.28.243.125:33979)
0809 22:43:08.408712 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.408717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.408737 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:43:08.408810 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.408814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.408920 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.409042 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.410045 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.410817 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.412547 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.413103 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.413222 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.417183 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.417188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.417191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.417229 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.420198 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.421339 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.421348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.421351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.421557 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.422048 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.422053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.422055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.422399 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.422579 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.423406 (+ 827us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.423716 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260809 22:43:08.424013 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.408233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41258)
0809 22:43:08.408460 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.408482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:08.408667 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:08.409097 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:08.409102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:08.409602 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:08.409840 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.409853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.410980 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.410986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.412362 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.412379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.413270 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.413277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.413483 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.414216 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:08.414240 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:08.416920 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:08.420408 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.420413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.420418 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.421069 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.421709 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.421714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.421719 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.421900 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.422558 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:08.422563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:08.422967 (+ 404us) client_negotiation.cc:769] Sending connection context
0809 22:43:08.423234 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 22:43:08.423620 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260809 22:43:08.425395 21598 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:43:08.426831 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.426390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44740)
0809 22:43:08.426599 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.426685 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260809 22:43:08.428035 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.427636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44752)
0809 22:43:08.427810 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.427907 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260809 22:43:08.428431 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.461508 21677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 2
I20260809 22:43:08.462847 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.462267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44768)
0809 22:43:08.462515 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.462637 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 22:43:08.464425 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.463959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44780)
0809 22:43:08.464190 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.464288 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260809 22:43:08.464776 21677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.479763 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 1
I20260809 22:43:08.481357 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.480628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44792)
0809 22:43:08.480856 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.481097 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 22:43:08.482656 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.482225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44796)
0809 22:43:08.482403 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.482483 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 22:43:08.482916 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.551195 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 2
I20260809 22:43:08.552925 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.552087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44798)
0809 22:43:08.552459 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.552661 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260809 22:43:08.554170 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.553760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44806)
0809 22:43:08.553961 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.554039 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260809 22:43:08.554397 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.556036 21677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 3
I20260809 22:43:08.557227 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.556743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44812)
0809 22:43:08.556947 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.557061 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 22:43:08.558465 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.557988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44816)
0809 22:43:08.558244 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.558328 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260809 22:43:08.558724 21677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.660681 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 3
I20260809 22:43:08.662163 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.661481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44832)
0809 22:43:08.661753 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.661934 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260809 22:43:08.663479 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.663067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44848)
0809 22:43:08.663256 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.663354 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 22:43:08.663779 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.701970 21677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 4
I20260809 22:43:08.703227 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.702710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44862)
0809 22:43:08.702977 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.703068 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 22:43:08.704308 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.703961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44874)
0809 22:43:08.704125 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.704205 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 22:43:08.704530 21677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
W20260809 22:43:08.738883 21685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 22:43:08.739317 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 22:43:08.739454 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 22:43:08.739605 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 22:43:08.748173 21685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 22:43:08.748314 21685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 22:43:08.748449 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 22:43:08.748615 21685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 22:43:08.748729 21685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 22:43:08.759603 21685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.243.84:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.243.126:32853,127.28.243.125:33979,127.28.243.124:42693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/krb5kdc/kudu_127.28.243.124.keytab
--principal=kudu/127.28.243.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.243.124:42693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.243.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 21:48:06 UTC on ccycloud.quasar-tdsnhu.root.comops.site
build id 4576819
ASAN enabled
I20260809 22:43:08.761067 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 22:43:08.762609 21685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 22:43:08.776122 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:08.776165 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 22:43:08.776883 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 22:43:08.777422 21685 server_base.cc:1034] running on GCE node
I20260809 22:43:08.778337 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 22:43:08.779961 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 22:43:08.781229 21685 hybrid_clock.cc:648] HybridClock initialized: now 1786315388781146 us; error 64 us; skew 500 ppm
I20260809 22:43:08.781611 21685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 22:43:08.785385 21685 init.cc:377] Logged in from keytab as kudu/127.28.243.124@KRBTEST.COM (short username kudu)
I20260809 22:43:08.787078 21685 webserver.cc:460] Webserver started at http://127.28.243.124:37029/ using document root <none> and password file <none>
I20260809 22:43:08.787828 21685 fs_manager.cc:359] Metadata directory not provided
I20260809 22:43:08.787971 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 22:43:08.788331 21685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 22:43:08.791685 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data/instance:
uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47"
format_stamp: "Formatted at 2026-08-09 22:43:08 on dist-test-slave-rgb1"
server_key: "2473d9bc5ff844d24d0f1063e7e038af"
server_key_iv: "822b3d5a6b0a34d80d458530b9b11a7b"
server_key_version: "encryptionkey@0"
I20260809 22:43:08.792519 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/wal/instance:
uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47"
format_stamp: "Formatted at 2026-08-09 22:43:08 on dist-test-slave-rgb1"
server_key: "2473d9bc5ff844d24d0f1063e7e038af"
server_key_iv: "822b3d5a6b0a34d80d458530b9b11a7b"
server_key_version: "encryptionkey@0"
I20260809 22:43:08.798048 21685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 22:43:08.801725 21700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 22:43:08.803225 21685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 22:43:08.803459 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data,/tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/wal
uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47"
format_stamp: "Formatted at 2026-08-09 22:43:08 on dist-test-slave-rgb1"
server_key: "2473d9bc5ff844d24d0f1063e7e038af"
server_key_iv: "822b3d5a6b0a34d80d458530b9b11a7b"
server_key_version: "encryptionkey@0"
I20260809 22:43:08.803702 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 22:43:08.820545 21685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 22:43:08.825896 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 22:43:08.826231 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 22:43:08.841071 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 4
I20260809 22:43:08.842437 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.841908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44880)
0809 22:43:08.842154 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.842249 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 22:43:08.843771 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.843229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44892)
0809 22:43:08.843440 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.843570 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260809 22:43:08.844084 21598 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.243.124:42693: Network error: Client connection negotiation failed: client connection to 127.28.243.124:42693: connect: Connection refused (error 111)
I20260809 22:43:08.848829 21685 rpc_server.cc:307] RPC server started. Bound to: 127.28.243.124:42693
I20260809 22:43:08.848850 21752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.243.124:42693 every 8 connection(s)
I20260809 22:43:08.850339 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/data/info.pb
I20260809 22:43:08.854499 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 22:43:08.858844 29645 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu as pid 21685
I20260809 22:43:08.859115 29645 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmuNQXR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786314994965430-29645-0/minicluster-data/master-2/wal/instance
I20260809 22:43:08.859587 29645 external_mini_cluster.cc:1428] Setting key 0e59f39675d26ef867253a49cdca1285
I20260809 22:43:08.863108 21754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:43:08.886240 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.867656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.28.243.126:32853)
0809 22:43:08.869047 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.869054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.869074 (+ 20us) server_negotiation.cc:407] Connection header received
0809 22:43:08.869138 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.869143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.869223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.869360 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.871312 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.872195 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.874383 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.874798 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.874924 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.879381 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.879387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.879390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.879426 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.882795 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.883778 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.883784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.883787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.883937 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.884374 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.884382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.884385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.884770 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.884976 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.885569 (+ 593us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.885807 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1253,"thread_start_us":1078,"threads_started":1}
I20260809 22:43:08.887409 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.867558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.126:32853 (local address 127.0.0.1:55884)
0809 22:43:08.868251 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.868294 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:08.868460 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:08.869455 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:08.869461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:08.870556 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:08.871070 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.871085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.872397 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.872414 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.874142 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.874155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.875067 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.875075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.875285 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.876040 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:08.876087 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:08.879091 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:08.882956 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.882968 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.882979 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.883568 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.884074 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.884080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.884083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.884251 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.884914 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:08.884924 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:08.885186 (+ 262us) client_negotiation.cc:769] Sending connection context
0809 22:43:08.885464 (+ 278us) client_negotiation.cc:240] Negotiation successful
0809 22:43:08.886019 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":76,"thread_start_us":297,"threads_started":1}
I20260809 22:43:08.887449 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.862074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.28.243.124:42693)
0809 22:43:08.863057 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.863070 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.863121 (+ 51us) server_negotiation.cc:407] Connection header received
0809 22:43:08.865528 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.865556 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.866091 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.866506 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.868687 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.870598 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.872182 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.873236 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.873417 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.878700 (+ 5283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.878717 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.878738 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.878799 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.882655 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.883568 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.883579 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.883583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.883822 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.884338 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.884347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.884354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.885150 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.885530 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.885981 (+ 451us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.886289 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":719,"thread_start_us":439,"threads_started":1}
I20260809 22:43:08.889775 21754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.126@KRBTEST.COM
I20260809 22:43:08.906387 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.890540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.125:33979 (local address 127.0.0.1:41260)
0809 22:43:08.890738 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.890761 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:08.890881 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:08.891332 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:08.891338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:08.891955 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:08.892201 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.892212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.893332 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.893340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.894660 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.894674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.895430 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.895438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.895677 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.896414 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:08.896444 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:08.899209 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:08.903499 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.903505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.903509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.903941 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.904407 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.904413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.904417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.904670 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.905266 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:08.905272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:08.905455 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 22:43:08.905630 (+ 175us) client_negotiation.cc:240] Negotiation successful
0809 22:43:08.905981 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 22:43:08.906420 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.890678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.28.243.125:33979)
0809 22:43:08.890856 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.890861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.890878 (+ 17us) server_negotiation.cc:407] Connection header received
0809 22:43:08.891096 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.891101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.891169 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.891257 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.892395 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.893171 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.894849 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.895260 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.895360 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.899435 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.899443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.899449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.899498 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.903355 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.904102 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.904109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.904113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.904275 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.904814 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.904822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.904825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.905134 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.905309 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.905734 (+ 425us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.905989 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:43:08.907454 21754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } has no permanent_uuid. Determining permanent_uuid...
I20260809 22:43:08.909770 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.891881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.28.243.126:32853)
0809 22:43:08.892048 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.892054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.892069 (+ 15us) server_negotiation.cc:407] Connection header received
0809 22:43:08.892113 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.892116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.892184 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.892262 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.893980 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.894675 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.896654 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.897073 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.897172 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.902555 (+ 5383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.902564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.902578 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.902632 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.906247 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.907297 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.907307 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.907322 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.907484 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.907994 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.908000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.908006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.908551 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.909012 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.909290 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.909508 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.124@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
Aug 09 22:43:08 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.243.125@KRBTEST.COM
I20260809 22:43:08.925112 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.908439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.28.243.124:42693)
0809 22:43:08.908747 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.908755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.908776 (+ 21us) server_negotiation.cc:407] Connection header received
0809 22:43:08.908845 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.908851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.908935 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.909051 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.910239 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.911316 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.913183 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.913620 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.913728 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.918007 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.918014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.918021 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.918062 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.921779 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.922796 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.922800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.922803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.922931 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.923484 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.923488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.923490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.923788 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.923961 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.924359 (+ 398us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.924708 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":176}
I20260809 22:43:08.925220 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.908208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44916)
0809 22:43:08.908417 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:08.908437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:08.908540 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:08.909103 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:08.909110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:08.909639 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:08.910081 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.910091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.911475 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.911479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.912999 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:08.913025 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.913783 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.913789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:08.914026 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.914756 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:08.914776 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:08.917777 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:08.921991 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.922018 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.922022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.922644 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.923131 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:08.923137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:08.923142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:08.923313 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:08.923932 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:08.923937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:08.924113 (+ 176us) client_negotiation.cc:769] Sending connection context
0809 22:43:08.924267 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 22:43:08.924859 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":250}
I20260809 22:43:08.929629 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:08.911980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.28.243.125:33979)
0809 22:43:08.912236 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:08.912241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:08.912260 (+ 19us) server_negotiation.cc:407] Connection header received
0809 22:43:08.912323 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:08.912327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:08.912399 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:08.912497 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:08.914074 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.915030 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.916901 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:08.917667 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:08.917821 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:08.923797 (+ 5976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:08.923802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:08.923805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:08.923840 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:08.926790 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.927540 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.927554 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.927557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.927719 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:08.928128 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:08.928133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:08.928136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:08.928416 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:08.928761 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:08.929039 (+ 278us) server_negotiation.cc:299] Negotiation successful
0809 22:43:08.929292 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 22:43:08.930783 21754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: Bootstrap starting.
I20260809 22:43:08.934978 21754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: Neither blocks nor log segments found. Creating new log.
I20260809 22:43:08.935417 29645 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 22:43:08.936415 21754 log.cc:824] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: Log is configured to *not* fsync() on all Append() calls
I20260809 22:43:08.939772 21754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: No bootstrap required, opened a new log
I20260809 22:43:08.938692 21569 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55900
I20260809 22:43:08.947929 21754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:08.948295 21754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597b78f5ac6e4f00ba57ed37b2c4ed47, State: Initialized, Role: FOLLOWER
I20260809 22:43:08.948915 21754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:08.949959 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:08.950373 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:08.950991 21754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 22:43:08.961495 21772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:43:08.961655 21772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:43:08.959205 21685 master_runner.cc:186] Error getting master registration for 127.28.243.126:32853: OK, instance_id { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" instance_seqno: 1786315387684317 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 22:43:08.972162 21685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 22:43:09.003803 21677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 5
Aug 09 22:43:09 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315388, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.125@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
I20260809 22:43:09.023331 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:09.005130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.28.243.124:42693)
0809 22:43:09.005374 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.005382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.005411 (+ 29us) server_negotiation.cc:407] Connection header received
0809 22:43:09.005479 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.005485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.005586 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.005708 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.007004 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.008185 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.010236 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.011075 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.011225 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:09.015043 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:09.015052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:09.015058 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:09.015105 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:09.019480 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.020669 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.020677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.020683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.020868 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.021486 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.021491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.021494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.021996 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:09.022174 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:09.022684 (+ 510us) server_negotiation.cc:299] Negotiation successful
0809 22:43:09.022936 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 22:43:09.023347 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:09.004686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44918)
0809 22:43:09.005041 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:09.005072 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:09.005240 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:09.005803 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:09.005809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:09.006518 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:09.006762 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:09.006776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.008432 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.008439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:09.010023 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:09.010046 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.011274 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.011283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:09.011530 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:09.012343 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:09.012365 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:09.014822 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:09.019743 (+ 4921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:09.019750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:09.019755 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:09.020407 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:09.021056 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:09.021063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:09.021067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:09.021325 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:09.022174 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:09.022180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:09.022383 (+ 203us) client_negotiation.cc:769] Sending connection context
0809 22:43:09.022545 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 22:43:09.022894 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 22:43:09.028527 21677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: Bootstrap starting.
I20260809 22:43:09.031869 21677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: Neither blocks nor log segments found. Creating new log.
I20260809 22:43:09.033093 21677 log.cc:824] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: Log is configured to *not* fsync() on all Append() calls
I20260809 22:43:09.036051 21677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: No bootstrap required, opened a new log
I20260809 22:43:09.043833 21677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:09.044286 21677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1f8101f8bd4fdda62460217fe6222c, State: Initialized, Role: FOLLOWER
I20260809 22:43:09.044850 21677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:09.046288 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.047240 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:09.047605 21677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [sys.catalog]: configured and running, proceeding with master startup.
W20260809 22:43:09.065551 21786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 22:43:09.065694 21786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 22:43:09.062670 21605 master_runner.cc:186] Error getting master registration for 127.28.243.126:32853: OK, instance_id { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" instance_seqno: 1786315387684317 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 22:43:09.076149 21605 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 22:43:09.078696 21706 tablet.cc:2369] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:43:09.121400 21598 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } attempt: 5
Aug 09 22:43:09 dist-test-slave-rgb1 krb5kdc[21513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786315387, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.243.126@KRBTEST.COM for kudu/127.28.243.124@KRBTEST.COM
I20260809 22:43:09.141786 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:09.122302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.243.124:42693 (local address 127.0.0.1:44926)
0809 22:43:09.122800 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0809 22:43:09.122823 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 22:43:09.122951 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 22:43:09.123446 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 22:43:09.123451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 22:43:09.124031 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0809 22:43:09.124270 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:09.124284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.125982 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.125989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:09.127690 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 22:43:09.127713 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.128878 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.128885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 22:43:09.129120 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:09.129929 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 22:43:09.129958 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 22:43:09.133387 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 22:43:09.138528 (+ 5141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:09.138543 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:09.138548 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:09.139152 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:09.139672 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 22:43:09.139680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 22:43:09.139685 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 22:43:09.139891 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 22:43:09.140643 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 22:43:09.140649 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 22:43:09.140841 (+ 192us) client_negotiation.cc:769] Sending connection context
0809 22:43:09.141011 (+ 170us) client_negotiation.cc:240] Negotiation successful
0809 22:43:09.141334 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344}
I20260809 22:43:09.142299 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:09.122823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.28.243.124:42693)
0809 22:43:09.123029 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.123036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.123062 (+ 26us) server_negotiation.cc:407] Connection header received
0809 22:43:09.123137 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.123143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.123258 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.123397 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.124490 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.125766 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.127872 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.128665 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.128793 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:09.133693 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:09.133702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:09.133708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:09.133760 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:09.138305 (+ 4545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.139301 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.139309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.139313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.139518 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.140017 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.140023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.140027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.140455 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:09.140643 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:09.141146 (+ 503us) server_negotiation.cc:299] Negotiation successful
0809 22:43:09.141421 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 22:43:09.149483 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9: Bootstrap starting.
I20260809 22:43:09.154338 21598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9: Neither blocks nor log segments found. Creating new log.
I20260809 22:43:09.155922 21598 log.cc:824] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9: Log is configured to *not* fsync() on all Append() calls
I20260809 22:43:09.160823 21598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9: No bootstrap required, opened a new log
I20260809 22:43:09.170447 21598 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:09.171059 21598 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc2bf5aa0227406684fbdb5f2f4fb0a9, State: Initialized, Role: FOLLOWER
I20260809 22:43:09.171868 21598 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:09.173499 21789 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.174263 21789 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:09.174774 21598 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 22:43:09.174922 21569 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 22:43:09.175179 21569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:43:09.179236 21569 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:09.182133 21648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5f1f8101f8bd4fdda62460217fe6222c"
I20260809 22:43:09.182742 21648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:43:09.187680 21648 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2bf5aa0227406684fbdb5f2f4fb0a9 in term 1.
I20260809 22:43:09.188683 21546 leader_election.cc:302] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1f8101f8bd4fdda62460217fe6222c, bc2bf5aa0227406684fbdb5f2f4fb0a9; no voters:
I20260809 22:43:09.189986 21569 leader_election.cc:288] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [CANDIDATE]: Term 1 election: Requested vote from peers 5f1f8101f8bd4fdda62460217fe6222c (127.28.243.125:33979), 597b78f5ac6e4f00ba57ed37b2c4ed47 (127.28.243.124:42693)
I20260809 22:43:09.191047 21727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47"
I20260809 22:43:09.191632 21727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [term 0 FOLLOWER]: Advancing to term 1
I20260809 22:43:09.195791 21727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc2bf5aa0227406684fbdb5f2f4fb0a9 in term 1.
I20260809 22:43:09.200775 21789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 22:43:09.208762 21789 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [term 1 LEADER]: Becoming Leader. State: Replica: bc2bf5aa0227406684fbdb5f2f4fb0a9, State: Running, Role: LEADER
I20260809 22:43:09.209759 21789 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } }
I20260809 22:43:09.222488 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc2bf5aa0227406684fbdb5f2f4fb0a9. Latest consensus state: current_term: 1 leader_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.222991 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:09.235391 21802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260809 22:43:09.250528 21626 tablet.cc:2369] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 22:43:09.254254 21802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 22:43:09.274154 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:09.223791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.28.243.126:32853)
0809 22:43:09.223976 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.223983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.231031 (+ 7048us) server_negotiation.cc:407] Connection header received
0809 22:43:09.232474 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.232481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.232606 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.232678 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.236893 (+ 4215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.238152 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.243774 (+ 5622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.244496 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.244620 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:09.258835 (+ 14215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:09.258844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:09.258849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:09.258896 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:09.263080 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.269556 (+ 6476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.269563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.269567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.269748 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.272069 (+ 2321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.272077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.272080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.272486 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:09.272817 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:09.273426 (+ 609us) server_negotiation.cc:299] Negotiation successful
0809 22:43:09.273766 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 22:43:09.274120 21812 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 22:43:09.292958 21727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [term 1 FOLLOWER]: Refusing update from remote peer bc2bf5aa0227406684fbdb5f2f4fb0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:43:09.294207 21795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:43:09.298169 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:09.231614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.28.243.125:33979)
0809 22:43:09.231803 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.231810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.241523 (+ 9713us) server_negotiation.cc:407] Connection header received
0809 22:43:09.242969 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.242976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.243069 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.243145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.253017 (+ 9872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.254293 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.267437 (+ 13144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.268165 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.268312 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:09.283192 (+ 14880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:09.283199 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:09.283206 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:09.283261 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:09.287898 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.289108 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.289119 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.289123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.289299 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.289867 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.289874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.289877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.290408 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:09.290726 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:09.297391 (+ 6665us) server_negotiation.cc:299] Negotiation successful
0809 22:43:09.297706 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 22:43:09.301841 21648 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [term 1 FOLLOWER]: Refusing update from remote peer bc2bf5aa0227406684fbdb5f2f4fb0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 22:43:09.303416 21795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 22:43:09.313848 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:09.242072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.28.243.124:42693)
0809 22:43:09.242269 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.242275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.249695 (+ 7420us) server_negotiation.cc:407] Connection header received
0809 22:43:09.251637 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.251643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.251739 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.251813 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.264236 (+ 12423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.265392 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.275228 (+ 9836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:09.276050 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:09.276191 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:09.291406 (+ 15215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 22:43:09.291414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 22:43:09.291418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 22:43:09.291480 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 22:43:09.296108 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.309353 (+ 13245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.309361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.309366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.309592 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 22:43:09.310900 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 22:43:09.310906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 22:43:09.310911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 22:43:09.311446 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 22:43:09.311768 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:09.313031 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0809 22:43:09.313374 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 22:43:09.314927 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc2bf5aa0227406684fbdb5f2f4fb0a9. Latest consensus state: current_term: 1 leader_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.315363 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:09.319232 21813 mvcc.cc:205] Tried to move back new op lower bound from 7316747834531561472 to 7316747834211061760. Current Snapshot: MvccSnapshot[applied={T|T < 7316747834531561472}]
I20260809 22:43:09.326943 21814 mvcc.cc:205] Tried to move back new op lower bound from 7316747834531561472 to 7316747834211061760. Current Snapshot: MvccSnapshot[applied={T|T < 7316747834531561472}]
I20260809 22:43:09.328434 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc2bf5aa0227406684fbdb5f2f4fb0a9. Latest consensus state: current_term: 1 leader_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.328847 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:09.333771 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.334062 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:09.345508 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.345896 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c [sys.catalog]: This master's current role is: FOLLOWER
I20260809 22:43:09.349747 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.350088 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:09.350536 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc2bf5aa0227406684fbdb5f2f4fb0a9" member_type: VOTER last_known_addr { host: "127.28.243.126" port: 32853 } } peers { permanent_uuid: "5f1f8101f8bd4fdda62460217fe6222c" member_type: VOTER last_known_addr { host: "127.28.243.125" port: 33979 } } peers { permanent_uuid: "597b78f5ac6e4f00ba57ed37b2c4ed47" member_type: VOTER last_known_addr { host: "127.28.243.124" port: 42693 } } }
I20260809 22:43:09.350838 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9 [sys.catalog]: This master's current role is: LEADER
I20260809 22:43:09.352161 21802 catalog_manager.cc:1347] Generated new cluster ID: a6066451351a41fba69623151c24e785
I20260809 22:43:09.352276 21802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 22:43:09.427028 21802 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 22:43:09.429154 21802 catalog_manager.cc:1504] Loading token signing keys...
I20260809 22:43:09.464341 21802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc2bf5aa0227406684fbdb5f2f4fb0a9: Generated new TSK 0
I20260809 22:43:09.467103 21802 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 22:43:09.589632 21759 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.586709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.28.243.126:32853)
0809 22:43:09.586965 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.586973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.587510 (+ 537us) server_negotiation.cc:407] Connection header received
0809 22:43:09.587826 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.587833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.587921 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.588008 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.589330 (+ 1322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55912: BlockingRecv error: recv got EOF from 127.0.0.1:55912 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260809 22:43:09.592137 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.587676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.28.243.125:33979)
0809 22:43:09.587902 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.587913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.590326 (+ 2413us) server_negotiation.cc:407] Connection header received
0809 22:43:09.590614 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.590621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.590706 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.590779 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.591803 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41298: BlockingRecv error: recv got EOF from 127.0.0.1:41298 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 22:43:09.595772 21757 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.593536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.28.243.124:42693)
0809 22:43:09.593752 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.593758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.593938 (+ 180us) server_negotiation.cc:407] Connection header received
0809 22:43:09.594125 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.594132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.594233 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.594377 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.595475 (+ 1098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44934: BlockingRecv error: recv got EOF from 127.0.0.1:44934 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260809 22:43:09.606416 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.603567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.28.243.125:33979)
0809 22:43:09.603789 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.603797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.604444 (+ 647us) server_negotiation.cc:407] Connection header received
0809 22:43:09.604802 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.604809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.604965 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.605103 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.606097 (+ 994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41312: BlockingRecv error: recv got EOF from 127.0.0.1:41312 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 22:43:09.613991 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.611391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.28.243.125:33979)
0809 22:43:09.611596 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.611603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.612213 (+ 610us) server_negotiation.cc:407] Connection header received
0809 22:43:09.612472 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.612478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.612631 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.612757 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.613746 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41320: BlockingRecv error: recv got EOF from 127.0.0.1:41320 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260809 22:43:09.622730 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.619781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.28.243.125:33979)
0809 22:43:09.620001 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.620009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.620788 (+ 779us) server_negotiation.cc:407] Connection header received
0809 22:43:09.621137 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.621144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.621269 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.621390 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.622453 (+ 1063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41322: BlockingRecv error: recv got EOF from 127.0.0.1:41322 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260809 22:43:09.633309 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.630643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.28.243.125:33979)
0809 22:43:09.630928 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.630936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.631546 (+ 610us) server_negotiation.cc:407] Connection header received
0809 22:43:09.631778 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.631785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.631889 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.632004 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.633050 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41336: BlockingRecv error: recv got EOF from 127.0.0.1:41336 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260809 22:43:09.649425 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.646457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.28.243.125:33979)
0809 22:43:09.646690 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.646698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.647408 (+ 710us) server_negotiation.cc:407] Connection header received
0809 22:43:09.647709 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.647718 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.647845 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.647977 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.649091 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41338: BlockingRecv error: recv got EOF from 127.0.0.1:41338 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260809 22:43:09.672937 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.670386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.28.243.125:33979)
0809 22:43:09.670610 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.670617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.671187 (+ 570us) server_negotiation.cc:407] Connection header received
0809 22:43:09.671380 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.671386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.671497 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.671615 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.672624 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41350: BlockingRecv error: recv got EOF from 127.0.0.1:41350 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260809 22:43:09.710731 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.708324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.28.243.125:33979)
0809 22:43:09.708607 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.708617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.709202 (+ 585us) server_negotiation.cc:407] Connection header received
0809 22:43:09.709432 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.709444 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.709524 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.709644 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.710465 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41352: BlockingRecv error: recv got EOF from 127.0.0.1:41352 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 22:43:09.781327 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.778856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.28.243.125:33979)
0809 22:43:09.779168 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.779175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.779587 (+ 412us) server_negotiation.cc:407] Connection header received
0809 22:43:09.779841 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.779846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.779924 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.780060 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.781101 (+ 1041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41364: BlockingRecv error: recv got EOF from 127.0.0.1:41364 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260809 22:43:09.915617 21682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 22:43:09.913326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.28.243.125:33979)
0809 22:43:09.913622 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:09.913631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:09.913939 (+ 308us) server_negotiation.cc:407] Connection header received
0809 22:43:09.914192 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:09.914198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:09.914281 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:09.914408 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 22:43:09.915344 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41370: BlockingRecv error: recv got EOF from 127.0.0.1:41370 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 22:43:09.964020 21772 catalog_manager.cc:1259] Loaded cluster ID: a6066451351a41fba69623151c24e785
I20260809 22:43:09.964128 21772 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: loading cluster ID for follower catalog manager: success
I20260809 22:43:09.966895 21772 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: acquiring CA information for follower catalog manager: success
I20260809 22:43:09.969503 21772 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 597b78f5ac6e4f00ba57ed37b2c4ed47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:43:10.068387 21786 catalog_manager.cc:1259] Loaded cluster ID: a6066451351a41fba69623151c24e785
I20260809 22:43:10.068778 21786 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: loading cluster ID for follower catalog manager: success
I20260809 22:43:10.072657 21786 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: acquiring CA information for follower catalog manager: success
I20260809 22:43:10.075479 21786 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f1f8101f8bd4fdda62460217fe6222c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 22:43:10.182248 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:10.175720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.28.243.125:33979)
0809 22:43:10.175969 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:10.175975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:10.176225 (+ 250us) server_negotiation.cc:407] Connection header received
0809 22:43:10.176460 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:10.176466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:10.176664 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:10.176798 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:10.177401 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.178153 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.179690 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.180155 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.180321 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:10.180931 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:10.181528 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:10.181717 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:10.181806 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 22:43:10.181922 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260809 22:43:10.196492 21890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:10.189535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.28.243.124:42693)
0809 22:43:10.190103 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:10.190109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:10.190314 (+ 205us) server_negotiation.cc:407] Connection header received
0809 22:43:10.190593 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:10.190597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:10.190791 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:10.190966 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:10.191460 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.192497 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.193981 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.194394 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.194493 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:10.195105 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:10.195698 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:10.195813 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:10.196006 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 22:43:10.196168 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":283,"threads_started":1}
I20260809 22:43:10.211347 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:10.204158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.28.243.125:33979)
0809 22:43:10.204370 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:10.204376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:10.204780 (+ 404us) server_negotiation.cc:407] Connection header received
0809 22:43:10.205246 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:10.205255 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:10.205368 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:10.205470 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:10.206216 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.207038 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.209259 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.209757 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.209882 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:10.210514 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:10.210632 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:10.210682 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:10.210934 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 22:43:10.211020 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 22:43:10.211678 21890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:10.204914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.28.243.124:42693)
0809 22:43:10.205080 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:10.205084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:10.205869 (+ 785us) server_negotiation.cc:407] Connection header received
0809 22:43:10.206044 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:10.206050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:10.206148 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:10.206238 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:10.206888 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.207830 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.209680 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.210302 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.210455 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:10.210902 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:10.211023 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:10.211104 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:10.211295 (+ 191us) server_negotiation.cc:299] Negotiation successful
0809 22:43:10.211400 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 22:43:10.211706 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 22:43:10.203456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.28.243.126:32853)
0809 22:43:10.203965 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0809 22:43:10.203971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 22:43:10.204126 (+ 155us) server_negotiation.cc:407] Connection header received
0809 22:43:10.204487 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 22:43:10.204497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 22:43:10.204797 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 22:43:10.205039 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 22:43:10.205793 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.206904 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.209149 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 22:43:10.209804 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 22:43:10.209945 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 22:43:10.210466 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 22:43:10.210947 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 22:43:10.211068 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 22:43:10.211201 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 22:43:10.211286 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":274,"threads_started":1}
I20260809 22:43:10.217581 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21527
I20260809 22:43:10.218653 21592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:43:10.331295 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 21527
I20260809 22:43:10.356851 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21605
I20260809 22:43:10.358136 21671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:43:10.470322 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 21605
I20260809 22:43:10.495312 29645 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu pid 21685
I20260809 22:43:10.496738 21748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 22:43:10.608229 29645 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmuNQXR/build/asan/bin/kudu with pid 21685
2026-08-09T22:43:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5063 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10148 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (395562 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST